2024-09-15T03:29:43.582 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T03:29:43.596 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T03:29:43.676 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905297 branch: wip-bharath9-testing-2024-09-13-1444-quincy description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} email: skanta@redhat.com first_in_suite: false job_id: '7905297' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath9-testing-2024-09-13-1444-quincy ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: d75f23820806da622f561ee5ee7551015285993f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: enable_coprs: - ceph/el9 flavor: default sha1: d75f23820806da622f561ee5ee7551015285993f workunit: branch: wip-bharath9-testing-2024-09-13-1444-quincy sha1: d75f23820806da622f561ee5ee7551015285993f owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 2126 sha1: d75f23820806da622f561ee5ee7551015285993f sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath9-testing-2024-09-13-1444-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d75f23820806da622f561ee5ee7551015285993f targets: smithi138.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhD2Xd/pfIDUtkDwbb4vujBEfM/deAXtUZBBfmB3lPImr79re33wYRyOj+TsubEPaCDujp6stAkK4Ykl1BF7IY= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_00:54:39 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T03:29:43.676 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa; will attempt to use it 2024-09-15T03:29:43.677 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d75f23820806da622f561ee5ee7551015285993f/qa/tasks 2024-09-15T03:29:43.677 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T03:29:43.679 INFO:teuthology.task.internal:Checking packages... 2024-09-15T03:29:43.700 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd75f23820806da622f561ee5ee7551015285993f' 2024-09-15T03:29:43.700 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T03:29:43.700 INFO:teuthology.packaging:ref: None 2024-09-15T03:29:43.700 INFO:teuthology.packaging:tag: None 2024-09-15T03:29:43.701 INFO:teuthology.packaging:branch: wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:29:43.701 INFO:teuthology.packaging:sha1: d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:29:43.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:29:43.849 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1643.gd75f2382 2024-09-15T03:29:43.851 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T03:29:43.857 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T03:29:43.858 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T03:29:43.866 INFO:teuthology.task.internal:Saving configuration 2024-09-15T03:29:43.874 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T03:29:43.881 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T03:29:43.905 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi138.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905297', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 03:24:44.229099', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAhD2Xd/pfIDUtkDwbb4vujBEfM/deAXtUZBBfmB3lPImr79re33wYRyOj+TsubEPaCDujp6stAkK4Ykl1BF7IY='} 2024-09-15T03:29:43.905 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T03:29:43.911 INFO:teuthology.task.internal:roles: ubuntu@smithi138.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-15T03:29:43.911 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T03:29:43.943 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb2d144bac0>, signals=[15]) 2024-09-15T03:29:43.943 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T03:29:43.949 INFO:teuthology.task.internal:Opening connections... 2024-09-15T03:29:43.950 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi138.front.sepia.ceph.com 2024-09-15T03:29:43.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:29:44.028 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T03:29:44.035 DEBUG:teuthology.orchestra.run.smithi138:> uname -m 2024-09-15T03:29:44.111 INFO:teuthology.orchestra.run.smithi138.stdout:x86_64 2024-09-15T03:29:44.111 DEBUG:teuthology.orchestra.run.smithi138:> cat /etc/os-release 2024-09-15T03:29:44.170 INFO:teuthology.orchestra.run.smithi138.stdout:NAME="CentOS Stream" 2024-09-15T03:29:44.170 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION="9" 2024-09-15T03:29:44.170 INFO:teuthology.orchestra.run.smithi138.stdout:ID="centos" 2024-09-15T03:29:44.170 INFO:teuthology.orchestra.run.smithi138.stdout:ID_LIKE="rhel fedora" 2024-09-15T03:29:44.170 INFO:teuthology.orchestra.run.smithi138.stdout:VERSION_ID="9" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:PLATFORM_ID="platform:el9" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:ANSI_COLOR="0;31" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:LOGO="fedora-logo-icon" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:HOME_URL="https://centos.org/" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-15T03:29:44.171 INFO:teuthology.orchestra.run.smithi138.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-15T03:29:44.172 INFO:teuthology.lock.ops:Updating smithi138.front.sepia.ceph.com on lock server 2024-09-15T03:29:44.190 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T03:29:44.198 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T03:29:44.207 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T03:29:44.207 DEBUG:teuthology.orchestra.run.smithi138:> test '!' -e /home/ubuntu/cephtest 2024-09-15T03:29:44.227 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T03:29:44.233 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T03:29:44.234 DEBUG:teuthology.orchestra.run.smithi138:> test -z $(ls -A /var/lib/ceph) 2024-09-15T03:29:44.337 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T03:29:44.370 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T03:29:44.386 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T03:29:44.386 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T03:29:44.386 DEBUG:teuthology.orchestra.run.smithi138:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T03:29:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:29:44.404 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-09-15T03:29:44.463 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-503.el9.x86_64 2024-09-15T03:29:44.463 INFO:teuthology.task.kernel:Running kernel on smithi138: 5.14.0-503.el9.x86_64 2024-09-15T03:29:44.463 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum install -y kernel 2024-09-15T03:29:46.847 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-15T03:29:48.540 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 36 MB/s | 20 MB 00:00 2024-09-15T03:29:53.008 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-15T03:29:54.533 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 113 kB/s | 19 kB 00:00 2024-09-15T03:29:55.630 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-15T03:30:01.152 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-15T03:30:03.117 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-15T03:30:03.117 INFO:teuthology.orchestra.run.smithi138.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-15T03:30:03.214 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:30:03.220 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:30:03.220 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-09-15T03:30:03.220 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:30:03.220 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2024-09-15T03:30:03.221 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:30:03.222 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 79 M 2024-09-15T03:30:03.222 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 126 M 2024-09-15T03:30:03.222 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:30:03.461 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 195 kB/s | 23 kB 00:00 2024-09-15T03:30:04.212 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 18 MB/s | 16 MB 00:00 2024-09-15T03:30:04.662 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 24 MB/s | 29 MB 00:01 2024-09-15T03:30:05.029 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 21 MB/s | 35 MB 00:01 2024-09-15T03:30:05.029 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:30:05.029 INFO:teuthology.orchestra.run.smithi138.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-15T03:30:05.709 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:30:06.077 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:30:06.077 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:30:07.029 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:30:07.029 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:30:09.463 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:30:10.325 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:30:10.414 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:13.204 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:30:13.349 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:18.187 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:30:18.249 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:30:23.096 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.431 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.471 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-15T03:31:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-15T03:31:01.848 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.660 INFO:teuthology.orchestra.run.smithi138.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-15T03:31:02.661 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:31:02.661 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:31:03.018 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:03.100 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:03.101 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:03.101 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-15T03:31:03.101 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T03:31:03.101 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T03:31:03.101 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T03:31:03.101 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi138.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T03:31:03.101 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q kernel | sort -rV | head -n 1 2024-09-15T03:31:03.159 INFO:teuthology.orchestra.run.smithi138.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-15T03:31:03.160 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-15T03:31:03.160 DEBUG:teuthology.orchestra.run.smithi138:> sudo rpm -qi grub2-tools 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Name : grub2-tools 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Epoch : 1 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Version : 2.06 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Release : 82.el9 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Architecture: x86_64 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-15T03:31:03.192 INFO:teuthology.orchestra.run.smithi138.stdout:Group : Unspecified 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Size : 8274670 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:License : GPLv3+ 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Packager : builder@centos.org 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Vendor : CentOS 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Summary : Support tools for GRUB. 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:Description : 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-15T03:31:03.193 INFO:teuthology.orchestra.run.smithi138.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-15T03:31:03.194 INFO:teuthology.orchestra.run.smithi138.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-15T03:31:03.194 INFO:teuthology.orchestra.run.smithi138.stdout:hardware devices. 2024-09-15T03:31:03.194 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:31:03.194 INFO:teuthology.orchestra.run.smithi138.stdout:This subpackage provides tools for support of all platforms. 2024-09-15T03:31:03.195 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-15T03:31:03.195 INFO:teuthology.task.kernel:Updating grub on smithi138 to boot 5.14.0-505.el9.x86_64 2024-09-15T03:31:03.195 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-15T03:31:03.882 INFO:teuthology.orchestra.run.smithi138.stderr:Generating grub configuration file ... 2024-09-15T03:31:05.240 INFO:teuthology.orchestra.run.smithi138.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-15T03:31:05.272 INFO:teuthology.orchestra.run.smithi138.stderr:done 2024-09-15T03:31:05.274 DEBUG:teuthology.orchestra.run.smithi138:> mktemp 2024-09-15T03:31:05.290 INFO:teuthology.orchestra.run.smithi138.stdout:/tmp/tmp.VykeCBm8QD 2024-09-15T03:31:05.290 DEBUG:teuthology.orchestra.run.smithi138:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VykeCBm8QD 2024-09-15T03:31:05.358 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod 0666 /tmp/tmp.VykeCBm8QD 2024-09-15T03:31:05.497 DEBUG:teuthology.orchestra.remote:smithi138:/tmp/tmp.VykeCBm8QD is 6KB 2024-09-15T03:31:05.546 DEBUG:teuthology.orchestra.run.smithi138:> rm -fr /tmp/tmp.VykeCBm8QD 2024-09-15T03:31:05.561 DEBUG:teuthology.orchestra.run.smithi138:> sudo /bin/ls /boot/loader/entries || true 2024-09-15T03:31:05.623 INFO:teuthology.orchestra.run.smithi138.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-15T03:31:05.623 INFO:teuthology.orchestra.run.smithi138.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-15T03:31:05.623 INFO:teuthology.orchestra.run.smithi138.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-15T03:31:05.623 INFO:teuthology.orchestra.run.smithi138.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-15T03:31:05.624 DEBUG:teuthology.orchestra.run.smithi138:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-15T03:31:05.791 DEBUG:teuthology.orchestra.run.smithi138:> sudo shutdown -r now 2024-09-15T03:31:35.821 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T03:31:35.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:31:35.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:31:54.301 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-09-15T03:32:03.310 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:32:03.311 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:06.365 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.138 2024-09-15T03:32:18.377 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:32:18.378 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:32:50.925 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-09-15T03:32:51.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:32:51.236 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-15T03:32:51.236 DEBUG:teuthology.orchestra.run.smithi138:> uname -r 2024-09-15T03:32:51.251 INFO:teuthology.orchestra.run.smithi138.stdout:5.14.0-505.el9.x86_64 2024-09-15T03:32:51.251 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-15T03:32:51.251 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T03:32:51.251 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-15T03:32:52.251 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T03:32:52.252 DEBUG:teuthology.orchestra.run.smithi138:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T03:32:52.333 INFO:teuthology.orchestra.run.smithi138.stdout:ttyS1 2024-09-15T03:32:52.354 DEBUG:teuthology.parallel:result is None 2024-09-15T03:32:52.355 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T03:32:52.367 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T03:32:52.368 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T03:32:52.386 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T03:32:52.430 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T03:32:52.444 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T03:32:52.445 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T03:32:52.479 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T03:32:52.491 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T03:32:52.491 DEBUG:teuthology.orchestra.run.smithi138:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T03:32:52.549 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:52.560 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:32:52.562 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T03:32:52.572 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T03:32:52.573 DEBUG:teuthology.orchestra.run.smithi138:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T03:32:52.629 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T03:32:52.645 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T03:32:52.646 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T03:32:52.686 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:32:52.771 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T03:32:52.828 DEBUG:teuthology.orchestra.run.smithi138:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:32:52.911 DEBUG:teuthology.orchestra.run.smithi138:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T03:32:52.952 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:32:52.953 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T03:32:53.018 DEBUG:teuthology.orchestra.run.smithi138:> sudo service rsyslog restart 2024-09-15T03:32:53.101 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T03:32:53.491 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T03:32:53.499 INFO:teuthology.task.internal:Starting timer... 2024-09-15T03:32:53.499 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T03:32:53.516 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T03:32:53.530 DEBUG:teuthology.orchestra.run.smithi138:> sudo service auditd rotate 2024-09-15T03:32:53.572 INFO:teuthology.orchestra.run.smithi138.stdout:Rotating logs: 2024-09-15T03:32:53.574 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T03:32:53.574 DEBUG:teuthology.orchestra.run.smithi138:> /usr/sbin/getenforce 2024-09-15T03:32:53.637 INFO:teuthology.orchestra.run.smithi138.stdout:Permissive 2024-09-15T03:32:53.637 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi138.front.sepia.ceph.com': 'permissive'} 2024-09-15T03:32:53.637 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T03:32:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:32:53.703 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T03:32:53.703 DEBUG:teuthology.orchestra.run.smithi138:> sudo /usr/sbin/setenforce permissive 2024-09-15T03:32:53.786 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T03:32:53.797 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-15T03:32:53.800 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-15T03:32:53.935 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T03:32:53.949 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T03:32:53.949 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi138.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T03:36:38.886 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi138.front.sepia.ceph.com')] 2024-09-15T03:36:38.887 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:36:38.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi138.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T03:36:38.966 DEBUG:teuthology.orchestra.run.smithi138:> true 2024-09-15T03:36:39.043 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi138.front.sepia.ceph.com' 2024-09-15T03:36:39.044 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T03:36:39.057 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T03:36:39.057 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T03:36:39.057 DEBUG:teuthology.orchestra.run.smithi138:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T03:36:39.111 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-15T03:36:39.126 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-15T03:36:39.150 INFO:teuthology.orchestra.run.smithi138.stderr:sudo: ntpd: command not found 2024-09-15T03:36:39.161 INFO:teuthology.orchestra.run.smithi138.stdout:506 Cannot talk to daemon 2024-09-15T03:36:39.176 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-15T03:36:39.190 INFO:teuthology.orchestra.run.smithi138.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-15T03:36:39.246 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:36:39.249 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T03:36:39.250 INFO:teuthology.run_tasks:Running task install... 2024-09-15T03:36:39.259 DEBUG:teuthology.task.install:project ceph 2024-09-15T03:36:39.259 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'}} 2024-09-15T03:36:39.259 DEBUG:teuthology.task.install:config {'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'sha1': 'd75f23820806da622f561ee5ee7551015285993f'} 2024-09-15T03:36:39.259 INFO:teuthology.task.install:Using flavor: default 2024-09-15T03:36:39.265 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T03:36:39.265 INFO:teuthology.task.install:extra packages: [] 2024-09-15T03:36:39.265 DEBUG:teuthology.orchestra.run.smithi138:> sudo dnf -y install 'dnf-command(copr)' 2024-09-15T03:36:39.843 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:45 ago on Sun 15 Sep 2024 03:35:54 AM UTC. 2024-09-15T03:36:39.931 INFO:teuthology.orchestra.run.smithi138.stdout:Package dnf-plugins-core-4.3.0-16.el9.noarch is already installed. 2024-09-15T03:36:39.974 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:36:39.975 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:36:39.975 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:36:40.041 DEBUG:teuthology.orchestra.run.smithi138:> sudo dnf -y copr enable ceph/el9 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:Enabling a Copr repository. Please note that this repository is not part 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:of the main distribution, and quality may vary. 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:The Fedora Project does not exercise any power over the contents of 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:this repository beyond the rules outlined in the Copr FAQ at 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:, 2024-09-15T03:36:40.213 INFO:teuthology.orchestra.run.smithi138.stderr:and packages are not held to any quality or security level. 2024-09-15T03:36:40.214 INFO:teuthology.orchestra.run.smithi138.stderr: 2024-09-15T03:36:40.214 INFO:teuthology.orchestra.run.smithi138.stderr:Please do not file bug reports about these packages in Fedora 2024-09-15T03:36:40.214 INFO:teuthology.orchestra.run.smithi138.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2024-09-15T03:36:40.273 INFO:teuthology.orchestra.run.smithi138.stdout:Repository successfully enabled. 2024-09-15T03:36:40.308 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': ['ceph/el9'], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd75f23820806da622f561ee5ee7551015285993f', 'tag': None, 'wait_for_package': False} 2024-09-15T03:36:40.308 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:36:40.466 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/ 2024-09-15T03:36:40.466 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1643.gd75f2382 2024-09-15T03:36:40.577 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-bharath9-testing-2024-09-13-1444-quincy/d75f23820806da622f561ee5ee7551015285993f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-15T03:36:40.577 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:36:40.577 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-15T03:36:40.603 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-15T03:36:40.603 DEBUG:teuthology.orchestra.run.smithi138:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/d75f23820806da622f561ee5ee7551015285993f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-15T03:36:40.669 DEBUG:teuthology.orchestra.run.smithi138:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-15T03:36:40.745 DEBUG:teuthology.orchestra.run.smithi138:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-15T03:36:40.809 INFO:teuthology.orchestra.run.smithi138.stdout:check_obsoletes = 1 2024-09-15T03:36:40.810 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum clean all 2024-09-15T03:36:41.043 INFO:teuthology.orchestra.run.smithi138.stdout:45 files removed 2024-09-15T03:36:41.069 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-radosgw 2024-09-15T03:36:41.637 INFO:teuthology.orchestra.run.smithi138.stdout:ceph packages for x86_64 277 kB/s | 77 kB 00:00 2024-09-15T03:36:41.929 INFO:teuthology.orchestra.run.smithi138.stdout:ceph noarch packages 43 kB/s | 11 kB 00:00 2024-09-15T03:36:42.205 INFO:teuthology.orchestra.run.smithi138.stdout:ceph source packages 7.5 kB/s | 1.8 kB 00:00 2024-09-15T03:36:42.406 INFO:teuthology.orchestra.run.smithi138.stdout:Copr repo for el9 owned by ceph 46 kB/s | 7.6 kB 00:00 2024-09-15T03:36:45.289 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - BaseOS 2.9 MB/s | 8.2 MB 00:02 2024-09-15T03:36:47.183 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-09-15T03:36:51.418 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-15T03:36:53.002 INFO:teuthology.orchestra.run.smithi138.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 19 kB 00:00 2024-09-15T03:36:53.923 INFO:teuthology.orchestra.run.smithi138.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-15T03:36:59.254 INFO:teuthology.orchestra.run.smithi138.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-15T03:37:01.780 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 14 M 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrading: 2024-09-15T03:37:01.796 INFO:teuthology.orchestra.run.smithi138.stdout: librados2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.1 M 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 5.0 M 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 22 M 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 637 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 458 k 2024-09-15T03:37:01.797 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2 x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 4.9 M 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 45 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 120 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 156 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 320 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 295 k 2024-09-15T03:37:01.798 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 100 k 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout:Install 22 Packages 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout:Upgrade 2 Packages 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:01.799 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 58 M 2024-09-15T03:37:01.800 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:02.456 INFO:teuthology.orchestra.run.smithi138.stdout:(1/24): ceph-base-17.2.7-1643.gd75f2382.el9.x86 11 MB/s | 5.0 MB 00:00 2024-09-15T03:37:02.506 INFO:teuthology.orchestra.run.smithi138.stdout:(2/24): ceph-selinux-17.2.7-1643.gd75f2382.el9. 505 kB/s | 25 kB 00:00 2024-09-15T03:37:02.606 INFO:teuthology.orchestra.run.smithi138.stdout:(3/24): libcephfs2-17.2.7-1643.gd75f2382.el9.x8 6.2 MB/s | 637 kB 00:00 2024-09-15T03:37:02.798 INFO:teuthology.orchestra.run.smithi138.stdout:(4/24): ceph-radosgw-17.2.7-1643.gd75f2382.el9. 18 MB/s | 14 MB 00:00 2024-09-15T03:37:02.832 INFO:teuthology.orchestra.run.smithi138.stdout:(5/24): libradosstriper1-17.2.7-1643.gd75f2382. 2.0 MB/s | 458 kB 00:00 2024-09-15T03:37:02.916 INFO:teuthology.orchestra.run.smithi138.stdout:(6/24): python3-ceph-argparse-17.2.7-1643.gd75f 541 kB/s | 45 kB 00:00 2024-09-15T03:37:03.033 INFO:teuthology.orchestra.run.smithi138.stdout:(7/24): librgw2-17.2.7-1643.gd75f2382.el9.x86_6 21 MB/s | 4.9 MB 00:00 2024-09-15T03:37:03.058 INFO:teuthology.orchestra.run.smithi138.stdout:(8/24): python3-ceph-common-17.2.7-1643.gd75f23 847 kB/s | 120 kB 00:00 2024-09-15T03:37:03.100 INFO:teuthology.orchestra.run.smithi138.stdout:(9/24): python3-cephfs-17.2.7-1643.gd75f2382.el 2.3 MB/s | 156 kB 00:00 2024-09-15T03:37:03.142 INFO:teuthology.orchestra.run.smithi138.stdout:(10/24): python3-rados-17.2.7-1643.gd75f2382.el 3.7 MB/s | 320 kB 00:00 2024-09-15T03:37:03.184 INFO:teuthology.orchestra.run.smithi138.stdout:(11/24): python3-rbd-17.2.7-1643.gd75f2382.el9. 3.4 MB/s | 295 kB 00:00 2024-09-15T03:37:03.210 INFO:teuthology.orchestra.run.smithi138.stdout:(12/24): python3-rgw-17.2.7-1643.gd75f2382.el9. 1.4 MB/s | 100 kB 00:00 2024-09-15T03:37:03.498 INFO:teuthology.orchestra.run.smithi138.stdout:(13/24): ceph-common-17.2.7-1643.gd75f2382.el9. 15 MB/s | 22 MB 00:01 2024-09-15T03:37:03.785 INFO:teuthology.orchestra.run.smithi138.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 79 kB/s | 45 kB 00:00 2024-09-15T03:37:03.961 INFO:teuthology.orchestra.run.smithi138.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 206 kB/s | 160 kB 00:00 2024-09-15T03:37:04.078 INFO:teuthology.orchestra.run.smithi138.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 362 kB/s | 42 kB 00:00 2024-09-15T03:37:04.103 INFO:teuthology.orchestra.run.smithi138.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 921 kB/s | 292 kB 00:00 2024-09-15T03:37:04.154 INFO:teuthology.orchestra.run.smithi138.stdout:(18/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.0 MB/s | 662 kB 00:00 2024-09-15T03:37:04.187 INFO:teuthology.orchestra.run.smithi138.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-09-15T03:37:04.213 INFO:teuthology.orchestra.run.smithi138.stdout:(20/24): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.2 MB/s | 308 kB 00:00 2024-09-15T03:37:04.238 INFO:teuthology.orchestra.run.smithi138.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 818 kB/s | 67 kB 00:00 2024-09-15T03:37:04.297 INFO:teuthology.orchestra.run.smithi138.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-15T03:37:04.475 INFO:teuthology.orchestra.run.smithi138.stdout:(23/24): librados2-17.2.7-1643.gd75f2382.el9.x8 12 MB/s | 3.1 MB 00:00 2024-09-15T03:37:04.531 INFO:teuthology.orchestra.run.smithi138.stdout:(24/24): librbd1-17.2.7-1643.gd75f2382.el9.x86_ 10 MB/s | 2.9 MB 00:00 2024-09-15T03:37:04.536 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:04.536 INFO:teuthology.orchestra.run.smithi138.stdout:Total 21 MB/s | 58 MB 00:02 2024-09-15T03:37:04.599 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:04.633 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:04.634 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:05.293 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:05.293 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:06.552 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:06.935 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-15T03:37:07.017 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:07.108 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:07.197 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:07.235 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:07.323 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:07.370 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-15T03:37:07.407 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-15T03:37:07.846 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-15T03:37:07.873 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:07.919 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 8/26 2024-09-15T03:37:07.948 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-15T03:37:07.990 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 10/26 2024-09-15T03:37:08.028 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:08.098 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:08.126 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:08.193 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 13/26 2024-09-15T03:37:08.247 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-15T03:37:08.285 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/26 2024-09-15T03:37:08.319 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-15T03:37:08.617 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-15T03:37:08.656 INFO:teuthology.orchestra.run.smithi138.stdout: Upgrading : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:08.755 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 18/26 2024-09-15T03:37:08.828 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 19/26 2024-09-15T03:37:08.860 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 20/26 2024-09-15T03:37:11.268 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:11.315 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:11.855 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 21/26 2024-09-15T03:37:11.895 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 22/26 2024-09-15T03:37:11.990 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-15T03:37:11.990 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-15T03:37:11.990 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:12.021 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /sys 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /proc 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /mnt 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /var/tmp 2024-09-15T03:37:22.135 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /home 2024-09-15T03:37:22.136 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /root 2024-09-15T03:37:22.136 INFO:teuthology.orchestra.run.smithi138.stdout:skipping the directory /tmp 2024-09-15T03:37:22.136 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:23.438 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:23.514 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 24/26 2024-09-15T03:37:23.514 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:23.515 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-15T03:37:23.515 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:23.515 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-15T03:37:23.515 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:23.516 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-15T03:37:23.600 INFO:teuthology.orchestra.run.smithi138.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 1/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 3/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 4/26 2024-09-15T03:37:24.906 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 5/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_6 6/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 7/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9. 8/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x8 9/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 10/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 11/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 12/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 13/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-15T03:37:24.907 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 19/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 23/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-15T03:37:24.908 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 25/26 2024-09-15T03:37:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-15T03:37:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:25.979 INFO:teuthology.orchestra.run.smithi138.stdout:Upgraded: 2024-09-15T03:37:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.979 INFO:teuthology.orchestra.run.smithi138.stdout: librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-radosgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-selinux-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: libradosstriper1-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: librgw2-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.980 INFO:teuthology.orchestra.run.smithi138.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-argparse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ceph-common-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:25.981 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:26.375 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-test 2024-09-15T03:37:26.937 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:27 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:27.080 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 48 M 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 163 k 2024-09-15T03:37:27.081 INFO:teuthology.orchestra.run.smithi138.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout:Install 4 Packages 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:27.082 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 48 M 2024-09-15T03:37:27.083 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 193 M 2024-09-15T03:37:27.083 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:27.329 INFO:teuthology.orchestra.run.smithi138.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.0 MB/s | 304 kB 00:00 2024-09-15T03:37:27.404 INFO:teuthology.orchestra.run.smithi138.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 852 kB/s | 64 kB 00:00 2024-09-15T03:37:27.512 INFO:teuthology.orchestra.run.smithi138.stdout:(3/4): libcephsqlite-17.2.7-1643.gd75f2382.el9. 495 kB/s | 163 kB 00:00 2024-09-15T03:37:30.221 INFO:teuthology.orchestra.run.smithi138.stdout:(4/4): ceph-test-17.2.7-1643.gd75f2382.el9.x86_ 16 MB/s | 48 MB 00:03 2024-09-15T03:37:30.223 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:30.223 INFO:teuthology.orchestra.run.smithi138.stdout:Total 15 MB/s | 48 MB 00:03 2024-09-15T03:37:30.251 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:30.270 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:30.270 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:30.635 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:30.636 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:31.165 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:31.231 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-15T03:37:31.271 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-15T03:37:31.298 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:35.790 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 3/4 2024-09-15T03:37:35.940 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 4/4 2024-09-15T03:37:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 1/4 2024-09-15T03:37:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2/4 2024-09-15T03:37:38.215 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-test-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: libcephsqlite-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-15T03:37:38.529 INFO:teuthology.orchestra.run.smithi138.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-15T03:37:38.530 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:38.530 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:38.847 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph 2024-09-15T03:37:39.435 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:40 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:39.578 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:39.580 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:39.580 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:37:39.580 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:39.580 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 6.5 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.1 M 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 1.4 M 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 240 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 3.9 M 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 15 M 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-15T03:37:39.581 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-15T03:37:39.582 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:39.583 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:39.584 INFO:teuthology.orchestra.run.smithi138.stdout:Install 34 Packages 2024-09-15T03:37:39.584 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:39.584 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 27 M 2024-09-15T03:37:39.584 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 96 M 2024-09-15T03:37:39.584 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:39.917 INFO:teuthology.orchestra.run.smithi138.stdout:(1/34): ceph-17.2.7-1643.gd75f2382.el9.x86_64.r 53 kB/s | 6.5 kB 00:00 2024-09-15T03:37:40.110 INFO:teuthology.orchestra.run.smithi138.stdout:(2/34): ceph-mds-17.2.7-1643.gd75f2382.el9.x86_ 6.6 MB/s | 2.1 MB 00:00 2024-09-15T03:37:40.152 INFO:teuthology.orchestra.run.smithi138.stdout:(3/34): ceph-mgr-17.2.7-1643.gd75f2382.el9.x86_ 3.9 MB/s | 1.4 MB 00:00 2024-09-15T03:37:40.219 INFO:teuthology.orchestra.run.smithi138.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1643.gd75f 3.5 MB/s | 240 kB 00:00 2024-09-15T03:37:40.277 INFO:teuthology.orchestra.run.smithi138.stdout:(5/34): ceph-mon-17.2.7-1643.gd75f2382.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-09-15T03:37:40.436 INFO:teuthology.orchestra.run.smithi138.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-09-15T03:37:40.486 INFO:teuthology.orchestra.run.smithi138.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-09-15T03:37:40.537 INFO:teuthology.orchestra.run.smithi138.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.6 MB/s | 135 kB 00:00 2024-09-15T03:37:40.570 INFO:teuthology.orchestra.run.smithi138.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 4.2 MB/s | 1.2 MB 00:00 2024-09-15T03:37:40.595 INFO:teuthology.orchestra.run.smithi138.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-15T03:37:40.621 INFO:teuthology.orchestra.run.smithi138.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-15T03:37:40.788 INFO:teuthology.orchestra.run.smithi138.stdout:(12/34): ceph-osd-17.2.7-1643.gd75f2382.el9.x86 23 MB/s | 15 MB 00:00 2024-09-15T03:37:40.847 INFO:teuthology.orchestra.run.smithi138.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 154 kB/s | 35 kB 00:00 2024-09-15T03:37:40.863 INFO:teuthology.orchestra.run.smithi138.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 643 kB/s | 172 kB 00:00 2024-09-15T03:37:40.889 INFO:teuthology.orchestra.run.smithi138.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-09-15T03:37:40.914 INFO:teuthology.orchestra.run.smithi138.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 613 kB/s | 77 kB 00:00 2024-09-15T03:37:40.940 INFO:teuthology.orchestra.run.smithi138.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 548 kB/s | 42 kB 00:00 2024-09-15T03:37:40.990 INFO:teuthology.orchestra.run.smithi138.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 431 kB/s | 43 kB 00:00 2024-09-15T03:37:41.007 INFO:teuthology.orchestra.run.smithi138.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 728 kB/s | 11 kB 00:00 2024-09-15T03:37:41.024 INFO:teuthology.orchestra.run.smithi138.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-09-15T03:37:41.049 INFO:teuthology.orchestra.run.smithi138.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-15T03:37:41.066 INFO:teuthology.orchestra.run.smithi138.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 552 kB/s | 23 kB 00:00 2024-09-15T03:37:41.092 INFO:teuthology.orchestra.run.smithi138.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 461 kB/s | 19 kB 00:00 2024-09-15T03:37:41.117 INFO:teuthology.orchestra.run.smithi138.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 389 kB/s | 20 kB 00:00 2024-09-15T03:37:41.143 INFO:teuthology.orchestra.run.smithi138.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-15T03:37:41.168 INFO:teuthology.orchestra.run.smithi138.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-15T03:37:41.193 INFO:teuthology.orchestra.run.smithi138.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-15T03:37:41.219 INFO:teuthology.orchestra.run.smithi138.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-15T03:37:41.244 INFO:teuthology.orchestra.run.smithi138.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-15T03:37:41.269 INFO:teuthology.orchestra.run.smithi138.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-15T03:37:41.294 INFO:teuthology.orchestra.run.smithi138.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-15T03:37:41.320 INFO:teuthology.orchestra.run.smithi138.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-15T03:37:41.345 INFO:teuthology.orchestra.run.smithi138.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-15T03:37:41.370 INFO:teuthology.orchestra.run.smithi138.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-15T03:37:41.376 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:41.377 INFO:teuthology.orchestra.run.smithi138.stdout:Total 15 MB/s | 27 MB 00:01 2024-09-15T03:37:41.503 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:41.540 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:41.541 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:41.940 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:41.940 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:43.233 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:43.307 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-15T03:37:43.339 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-15T03:37:43.372 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-15T03:37:43.464 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-15T03:37:43.542 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-15T03:37:43.588 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-15T03:37:43.623 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-15T03:37:43.653 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-15T03:37:43.688 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-15T03:37:43.721 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-15T03:37:43.762 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-15T03:37:43.797 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-15T03:37:43.832 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-15T03:37:43.907 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-15T03:37:43.946 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:37:44.020 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-15T03:37:44.175 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-15T03:37:44.263 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-15T03:37:44.325 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-15T03:37:44.364 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-15T03:37:44.414 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-15T03:37:44.469 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-15T03:37:44.656 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-15T03:37:44.702 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-15T03:37:44.780 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-15T03:37:44.870 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-15T03:37:44.910 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-15T03:37:45.024 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-15T03:37:45.180 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 29/34 2024-09-15T03:37:45.211 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:37:45.237 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 30/34 2024-09-15T03:37:45.237 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:45.237 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-15T03:37:45.237 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:37:45.238 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-15T03:37:45.238 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:46.553 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:37:46.580 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 31/34 2024-09-15T03:37:46.581 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:46.581 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-15T03:37:46.581 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:37:46.581 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-15T03:37:46.581 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:47.138 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 32/34 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-15T03:37:47.162 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:47.372 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 33/34 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-15T03:37:47.396 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:47.468 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 34/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 1/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 3/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 4/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 5/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9. 6/34 2024-09-15T03:37:49.065 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-15T03:37:49.066 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-15T03:37:49.067 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-15T03:37:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-15T03:37:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:49.943 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:37:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.943 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mds-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-modules-core-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mon-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-osd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-15T03:37:49.944 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-ply-3.11-14.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-15T03:37:49.945 INFO:teuthology.orchestra.run.smithi138.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:49.946 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:50.371 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-base 2024-09-15T03:37:50.939 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:51 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:51.029 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-base-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:37:51.070 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:51.071 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:37:51.071 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:51.139 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install cephadm 2024-09-15T03:37:51.701 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:52 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:51.834 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:51.834 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:51.834 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 83 k 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 83 k 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 375 k 2024-09-15T03:37:51.835 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:52.024 INFO:teuthology.orchestra.run.smithi138.stdout:cephadm-17.2.7-1643.gd75f2382.el9.noarch.rpm 440 kB/s | 83 kB 00:00 2024-09-15T03:37:52.024 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:52.024 INFO:teuthology.orchestra.run.smithi138.stdout:Total 436 kB/s | 83 kB 00:00 2024-09-15T03:37:52.025 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:52.030 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:52.030 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:52.032 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:52.032 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:52.111 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:52.523 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:37:52.611 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:37:52.954 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:37:53.208 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:37:53.208 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:53.209 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:37:53.209 INFO:teuthology.orchestra.run.smithi138.stdout: cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:37:53.209 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:53.209 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:53.430 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-immutable-object-cache 2024-09-15T03:37:53.997 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:54 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:54.135 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:54.136 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:54.136 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-09-15T03:37:54.136 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:54.136 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 137 k 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 137 k 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 379 k 2024-09-15T03:37:54.137 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:54.351 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-immutable-object-cache-17.2.7-1643.gd75f23 644 kB/s | 137 kB 00:00 2024-09-15T03:37:54.351 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:54.351 INFO:teuthology.orchestra.run.smithi138.stdout:Total 639 kB/s | 137 kB 00:00 2024-09-15T03:37:54.352 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:54.359 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:54.359 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:54.416 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:54.416 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:54.561 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:54.610 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:37:54.635 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:37:54.635 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:37:54.635 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-15T03:37:54.635 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382. 1/1 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-immutable-object-cache-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:55.550 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:55.799 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr 2024-09-15T03:37:56.376 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:57 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:56.468 INFO:teuthology.orchestra.run.smithi138.stdout:Package ceph-mgr-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:37:56.510 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:56.511 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:37:56.511 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:37:56.582 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-dashboard 2024-09-15T03:37:57.141 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:00:58 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:37:57.275 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:37:57.276 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:57.276 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:37:57.276 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:57.276 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 3.4 M 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 24 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 15 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout:Installing weak dependencies: 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:37:57.277 INFO:teuthology.orchestra.run.smithi138.stdout:Install 7 Packages 2024-09-15T03:37:57.278 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:37:57.278 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 3.7 M 2024-09-15T03:37:57.278 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 80 M 2024-09-15T03:37:57.278 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:37:57.491 INFO:teuthology.orchestra.run.smithi138.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1643.gd75 158 kB/s | 24 kB 00:00 2024-09-15T03:37:57.517 INFO:teuthology.orchestra.run.smithi138.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1643.gd75f 81 kB/s | 15 kB 00:00 2024-09-15T03:37:57.558 INFO:teuthology.orchestra.run.smithi138.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-15T03:37:57.575 INFO:teuthology.orchestra.run.smithi138.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 2.0 MB/s | 31 kB 00:00 2024-09-15T03:37:57.592 INFO:teuthology.orchestra.run.smithi138.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 541 kB/s | 41 kB 00:00 2024-09-15T03:37:57.626 INFO:teuthology.orchestra.run.smithi138.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-15T03:37:57.743 INFO:teuthology.orchestra.run.smithi138.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1643.gd75f2382 8.3 MB/s | 3.4 MB 00:00 2024-09-15T03:37:57.747 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:37:57.747 INFO:teuthology.orchestra.run.smithi138.stdout:Total 7.8 MB/s | 3.7 MB 00:00 2024-09-15T03:37:57.747 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:37:57.755 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:37:57.756 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:37:57.839 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:37:57.839 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:37:58.141 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:37:58.211 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-15T03:37:58.253 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-15T03:37:58.308 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-15T03:37:58.349 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-15T03:37:58.378 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 5/7 2024-09-15T03:37:59.368 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 6/7 2024-09-15T03:37:59.416 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 7/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9. 1/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarc 2/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.n 3/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-15T03:38:00.516 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-grafana-dashboards-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-dashboard-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-prometheus-alerts-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-15T03:38:00.802 INFO:teuthology.orchestra.run.smithi138.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-15T03:38:00.803 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:00.803 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:01.023 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-15T03:38:01.581 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:02 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:01.713 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:01.714 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 7.4 M 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:38:01.715 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-15T03:38:01.716 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:01.717 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:01.718 INFO:teuthology.orchestra.run.smithi138.stdout:======================================================================================= 2024-09-15T03:38:01.718 INFO:teuthology.orchestra.run.smithi138.stdout:Install 12 Packages 2024-09-15T03:38:01.718 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:01.719 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 38 M 2024-09-15T03:38:01.719 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 212 M 2024-09-15T03:38:01.719 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:02.253 INFO:teuthology.orchestra.run.smithi138.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-15T03:38:02.303 INFO:teuthology.orchestra.run.smithi138.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 654 kB/s | 33 kB 00:00 2024-09-15T03:38:02.370 INFO:teuthology.orchestra.run.smithi138.stdout:(3/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-15T03:38:02.420 INFO:teuthology.orchestra.run.smithi138.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-15T03:38:02.471 INFO:teuthology.orchestra.run.smithi138.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 768 kB/s | 38 kB 00:00 2024-09-15T03:38:02.938 INFO:teuthology.orchestra.run.smithi138.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.7 MB/s | 3.0 MB 00:00 2024-09-15T03:38:03.030 INFO:teuthology.orchestra.run.smithi138.stdout:(7/12): ceph-mgr-diskprediction-local-17.2.7-16 7.8 MB/s | 7.4 MB 00:00 2024-09-15T03:38:03.064 INFO:teuthology.orchestra.run.smithi138.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-09-15T03:38:03.148 INFO:teuthology.orchestra.run.smithi138.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-15T03:38:03.440 INFO:teuthology.orchestra.run.smithi138.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.1 MB/s | 4.9 MB 00:00 2024-09-15T03:38:04.273 INFO:teuthology.orchestra.run.smithi138.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-09-15T03:38:05.440 INFO:teuthology.orchestra.run.smithi138.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-09-15T03:38:05.441 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:05.441 INFO:teuthology.orchestra.run.smithi138.stdout:Total 10 MB/s | 38 MB 00:03 2024-09-15T03:38:05.791 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:05.815 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:05.815 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:06.313 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:06.313 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:07.087 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:07.171 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-15T03:38:07.222 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:07.323 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:07.365 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-15T03:38:07.548 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-15T03:38:07.589 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-15T03:38:07.665 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-15T03:38:08.313 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-15T03:38:08.428 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-15T03:38:09.667 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:10.696 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-15T03:38:10.727 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:13.544 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 12/12 2024-09-15T03:38:13.544 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2 1/12 2024-09-15T03:38:13.544 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-15T03:38:13.544 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-15T03:38:13.544 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-15T03:38:13.545 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-15T03:38:13.951 INFO:teuthology.orchestra.run.smithi138.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:13.952 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:14.401 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-rook 2024-09-15T03:38:14.980 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:15 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:15.115 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 47 k 2024-09-15T03:38:15.116 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-15T03:38:15.117 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout:Install 13 Packages 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.2 M 2024-09-15T03:38:15.118 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 26 M 2024-09-15T03:38:15.119 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:15.342 INFO:teuthology.orchestra.run.smithi138.stdout:(1/13): ceph-mgr-rook-17.2.7-1643.gd75f2382.el9 277 kB/s | 47 kB 00:00 2024-09-15T03:38:15.684 INFO:teuthology.orchestra.run.smithi138.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 37 kB/s | 19 kB 00:00 2024-09-15T03:38:15.743 INFO:teuthology.orchestra.run.smithi138.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 45 kB/s | 26 kB 00:00 2024-09-15T03:38:16.085 INFO:teuthology.orchestra.run.smithi138.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 299 kB/s | 222 kB 00:00 2024-09-15T03:38:16.402 INFO:teuthology.orchestra.run.smithi138.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-15T03:38:16.444 INFO:teuthology.orchestra.run.smithi138.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 795 kB/s | 32 kB 00:00 2024-09-15T03:38:16.469 INFO:teuthology.orchestra.run.smithi138.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 563 kB/s | 14 kB 00:00 2024-09-15T03:38:16.494 INFO:teuthology.orchestra.run.smithi138.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.6 MB/s | 220 kB 00:00 2024-09-15T03:38:16.528 INFO:teuthology.orchestra.run.smithi138.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 355 kB/s | 279 kB 00:00 2024-09-15T03:38:16.578 INFO:teuthology.orchestra.run.smithi138.stdout:(10/13): python3-kubernetes-26.1.0-2.el9.noarch 12 MB/s | 1.0 MB 00:00 2024-09-15T03:38:16.604 INFO:teuthology.orchestra.run.smithi138.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-15T03:38:16.629 INFO:teuthology.orchestra.run.smithi138.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-09-15T03:38:16.738 INFO:teuthology.orchestra.run.smithi138.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 82 kB/s | 54 kB 00:00 2024-09-15T03:38:16.741 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:16.742 INFO:teuthology.orchestra.run.smithi138.stdout:Total 1.4 MB/s | 2.2 MB 00:01 2024-09-15T03:38:16.809 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:16.824 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:16.825 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:16.913 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:16.914 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:17.433 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:17.603 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-15T03:38:17.665 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-15T03:38:17.720 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-15T03:38:17.758 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-15T03:38:17.816 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-15T03:38:17.919 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-15T03:38:18.029 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-15T03:38:18.093 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-15T03:38:18.815 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-15T03:38:18.866 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-15T03:38:18.908 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-15T03:38:18.984 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-15T03:38:19.012 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:19.777 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 13/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 1/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-15T03:38:19.778 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-15T03:38:20.113 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-15T03:38:20.113 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-rook-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-15T03:38:20.114 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:20.115 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:20.361 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-mgr-cephadm 2024-09-15T03:38:20.924 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:21 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:21.058 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:21.058 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:21.058 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 124 k 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-15T03:38:21.059 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:21.060 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:21.061 INFO:teuthology.orchestra.run.smithi138.stdout:Install 6 Packages 2024-09-15T03:38:21.061 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:21.062 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 7.0 M 2024-09-15T03:38:21.062 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 33 M 2024-09-15T03:38:21.062 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:21.669 INFO:teuthology.orchestra.run.smithi138.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-09-15T03:38:21.695 INFO:teuthology.orchestra.run.smithi138.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1643.gd75f2382.e 624 kB/s | 124 kB 00:00 2024-09-15T03:38:21.737 INFO:teuthology.orchestra.run.smithi138.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.4 MB/s | 58 kB 00:00 2024-09-15T03:38:21.762 INFO:teuthology.orchestra.run.smithi138.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-15T03:38:21.804 INFO:teuthology.orchestra.run.smithi138.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 4.0 MB/s | 548 kB 00:00 2024-09-15T03:38:21.913 INFO:teuthology.orchestra.run.smithi138.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 6.0 MB 00:00 2024-09-15T03:38:21.915 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:21.915 INFO:teuthology.orchestra.run.smithi138.stdout:Total 8.2 MB/s | 7.0 MB 00:00 2024-09-15T03:38:21.989 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:22.000 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:22.000 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:22.097 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:22.099 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:22.396 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:22.564 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-15T03:38:22.615 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-15T03:38:23.137 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-15T03:38:23.243 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-15T03:38:23.299 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-15T03:38:23.328 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:24.097 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 6/6 2024-09-15T03:38:24.098 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 1/6 2024-09-15T03:38:24.098 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-15T03:38:24.098 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-15T03:38:24.098 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-15T03:38:24.098 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-15T03:38:24.369 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-15T03:38:24.369 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-mgr-cephadm-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:24.370 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:24.654 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-fuse 2024-09-15T03:38:25.214 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:26 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:25.347 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 813 k 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout:Installing dependencies: 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-15T03:38:25.348 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout:Install 2 Packages 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 893 k 2024-09-15T03:38:25.349 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 2.6 M 2024-09-15T03:38:25.350 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:25.534 INFO:teuthology.orchestra.run.smithi138.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 661 kB/s | 80 kB 00:00 2024-09-15T03:38:25.776 INFO:teuthology.orchestra.run.smithi138.stdout:(2/2): ceph-fuse-17.2.7-1643.gd75f2382.el9.x86_ 2.2 MB/s | 813 kB 00:00 2024-09-15T03:38:25.777 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:25.777 INFO:teuthology.orchestra.run.smithi138.stdout:Total 2.0 MB/s | 893 kB 00:00 2024-09-15T03:38:25.788 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:25.809 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:25.809 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:25.887 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:25.887 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:26.074 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:26.219 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-15T03:38:26.293 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:27.001 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2/2 2024-09-15T03:38:27.001 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/2 2024-09-15T03:38:27.294 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-15T03:38:27.295 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:27.295 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:27.295 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-15T03:38:27.295 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:27.295 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:27.613 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install ceph-volume 2024-09-15T03:38:28.191 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:29 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:28.325 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repository Size 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume noarch 2:17.2.7-1643.gd75f2382.el9 ceph-noarch 270 k 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:28.326 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:28.327 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 270 k 2024-09-15T03:38:28.327 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 1.3 M 2024-09-15T03:38:28.327 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:28.601 INFO:teuthology.orchestra.run.smithi138.stdout:ceph-volume-17.2.7-1643.gd75f2382.el9.noarch.rp 987 kB/s | 270 kB 00:00 2024-09-15T03:38:28.601 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:28.602 INFO:teuthology.orchestra.run.smithi138.stdout:Total 981 kB/s | 270 kB 00:00 2024-09-15T03:38:28.602 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:28.610 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:28.610 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:28.639 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:28.639 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:28.852 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:28.928 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:28.945 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:28.945 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:28.945 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-15T03:38:28.945 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:29.937 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 1/1 2024-09-15T03:38:29.937 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:29.937 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:29.938 INFO:teuthology.orchestra.run.smithi138.stdout: ceph-volume-2:17.2.7-1643.gd75f2382.el9.noarch 2024-09-15T03:38:29.938 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:29.938 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:30.244 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados-devel 2024-09-15T03:38:30.815 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:31 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:30.951 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 126 k 2024-09-15T03:38:30.952 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 126 k 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 448 k 2024-09-15T03:38:30.953 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:31.159 INFO:teuthology.orchestra.run.smithi138.stdout:librados-devel-17.2.7-1643.gd75f2382.el9.x86_64 612 kB/s | 126 kB 00:00 2024-09-15T03:38:31.160 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:31.160 INFO:teuthology.orchestra.run.smithi138.stdout:Total 608 kB/s | 126 kB 00:00 2024-09-15T03:38:31.160 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:31.167 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:31.168 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:31.228 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:31.228 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:31.381 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:31.481 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:31.881 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:32.163 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:32.163 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:32.163 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:32.163 INFO:teuthology.orchestra.run.smithi138.stdout: librados-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:32.163 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:32.164 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:32.472 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs2 2024-09-15T03:38:33.053 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:33.147 INFO:teuthology.orchestra.run.smithi138.stdout:Package libcephfs2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:33.191 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:33.192 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:33.193 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:33.259 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install libcephfs-devel 2024-09-15T03:38:33.820 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:34 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:33.951 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 25 k 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:33.952 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:33.953 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:33.953 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 25 k 2024-09-15T03:38:33.953 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 110 k 2024-09-15T03:38:33.953 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:34.091 INFO:teuthology.orchestra.run.smithi138.stdout:libcephfs-devel-17.2.7-1643.gd75f2382.el9.x86_6 180 kB/s | 25 kB 00:00 2024-09-15T03:38:34.092 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:34.092 INFO:teuthology.orchestra.run.smithi138.stdout:Total 178 kB/s | 25 kB 00:00 2024-09-15T03:38:34.092 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:34.095 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:34.096 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:34.097 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:34.097 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:34.184 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:34.287 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:34.662 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:34.894 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:34.894 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:34.894 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:34.894 INFO:teuthology.orchestra.run.smithi138.stdout: libcephfs-devel-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:34.895 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:34.895 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:35.091 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librados2 2024-09-15T03:38:35.678 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:36 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:35.780 INFO:teuthology.orchestra.run.smithi138.stdout:Package librados2-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:35.823 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:35.824 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:35.824 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:35.900 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install librbd1 2024-09-15T03:38:36.469 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:37 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:36.559 INFO:teuthology.orchestra.run.smithi138.stdout:Package librbd1-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:36.602 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:36.603 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:36.603 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:36.674 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rados 2024-09-15T03:38:37.248 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:38 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:37.339 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rados-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:37.381 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:37.382 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:37.383 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:37.450 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rgw 2024-09-15T03:38:38.021 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:38.111 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rgw-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:38.155 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:38.156 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:38.156 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:38.233 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-cephfs 2024-09-15T03:38:38.801 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:39 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:38.896 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-cephfs-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:38.935 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:38.936 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:38.936 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:39.010 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install python3-rbd 2024-09-15T03:38:39.580 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:40 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:39.670 INFO:teuthology.orchestra.run.smithi138.stdout:Package python3-rbd-2:17.2.7-1643.gd75f2382.el9.x86_64 is already installed. 2024-09-15T03:38:39.713 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:39.716 INFO:teuthology.orchestra.run.smithi138.stdout:Nothing to do. 2024-09-15T03:38:39.716 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:39.785 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-fuse 2024-09-15T03:38:40.362 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:41 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:40.497 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 82 k 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:40.498 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:40.499 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 82 k 2024-09-15T03:38:40.499 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 222 k 2024-09-15T03:38:40.499 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:40.721 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-fuse-17.2.7-1643.gd75f2382.el9.x86_64.rpm 369 kB/s | 82 kB 00:00 2024-09-15T03:38:40.721 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:40.721 INFO:teuthology.orchestra.run.smithi138.stdout:Total 366 kB/s | 82 kB 00:00 2024-09-15T03:38:40.722 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:40.728 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:40.728 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:40.791 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:40.792 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:40.941 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:41.051 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:41.445 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-fuse-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:41.722 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:42.072 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-mirror 2024-09-15T03:38:42.689 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:43 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:42.828 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout: Package Arch Version Repo Size 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 2.9 M 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:42.829 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:42.830 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 2.9 M 2024-09-15T03:38:42.830 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 12 M 2024-09-15T03:38:42.830 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:43.161 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-mirror-17.2.7-1643.gd75f2382.el9.x86_64.rpm 8.9 MB/s | 2.9 MB 00:00 2024-09-15T03:38:43.162 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:43.163 INFO:teuthology.orchestra.run.smithi138.stdout:Total 8.8 MB/s | 2.9 MB 00:00 2024-09-15T03:38:43.163 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:43.172 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:43.172 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:43.251 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:43.251 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:43.658 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:43.713 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-15T03:38:43.739 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-mirror-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:44.869 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:45.159 DEBUG:teuthology.orchestra.run.smithi138:> sudo yum -y install rbd-nbd 2024-09-15T03:38:45.745 INFO:teuthology.orchestra.run.smithi138.stdout:Last metadata expiration check: 0:01:46 ago on Sun 15 Sep 2024 03:36:59 AM UTC. 2024-09-15T03:38:45.878 INFO:teuthology.orchestra.run.smithi138.stdout:Dependencies resolved. 2024-09-15T03:38:45.879 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: Package Architecture Version Repository Size 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Installing: 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd x86_64 2:17.2.7-1643.gd75f2382.el9 ceph 165 k 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction Summary 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:================================================================================ 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Install 1 Package 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Total download size: 165 k 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Installed size: 477 k 2024-09-15T03:38:45.880 INFO:teuthology.orchestra.run.smithi138.stdout:Downloading Packages: 2024-09-15T03:38:46.093 INFO:teuthology.orchestra.run.smithi138.stdout:rbd-nbd-17.2.7-1643.gd75f2382.el9.x86_64.rpm 776 kB/s | 165 kB 00:00 2024-09-15T03:38:46.094 INFO:teuthology.orchestra.run.smithi138.stdout:-------------------------------------------------------------------------------- 2024-09-15T03:38:46.094 INFO:teuthology.orchestra.run.smithi138.stdout:Total 770 kB/s | 165 kB 00:00 2024-09-15T03:38:46.094 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction check 2024-09-15T03:38:46.101 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction check succeeded. 2024-09-15T03:38:46.102 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction test 2024-09-15T03:38:46.166 INFO:teuthology.orchestra.run.smithi138.stdout:Transaction test succeeded. 2024-09-15T03:38:46.166 INFO:teuthology.orchestra.run.smithi138.stdout:Running transaction 2024-09-15T03:38:46.315 INFO:teuthology.orchestra.run.smithi138.stdout: Preparing : 1/1 2024-09-15T03:38:46.415 INFO:teuthology.orchestra.run.smithi138.stdout: Installing : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:46.880 INFO:teuthology.orchestra.run.smithi138.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:47.185 INFO:teuthology.orchestra.run.smithi138.stdout: Verifying : rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 1/1 2024-09-15T03:38:47.185 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:47.186 INFO:teuthology.orchestra.run.smithi138.stdout:Installed: 2024-09-15T03:38:47.186 INFO:teuthology.orchestra.run.smithi138.stdout: rbd-nbd-2:17.2.7-1643.gd75f2382.el9.x86_64 2024-09-15T03:38:47.186 INFO:teuthology.orchestra.run.smithi138.stdout: 2024-09-15T03:38:47.186 INFO:teuthology.orchestra.run.smithi138.stdout:Complete! 2024-09-15T03:38:47.485 DEBUG:teuthology.parallel:result is None 2024-09-15T03:38:47.486 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:38:47.631 DEBUG:teuthology.orchestra.run.smithi138:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-15T03:38:47.652 INFO:teuthology.orchestra.run.smithi138.stdout:17.2.7-1643.gd75f2382.el9 2024-09-15T03:38:47.653 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1643.gd75f2382.el9 2024-09-15T03:38:47.653 INFO:teuthology.task.install:The correct ceph version 17.2.7-1643.gd75f2382 is installed. 2024-09-15T03:38:47.655 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T03:38:47.655 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:38:47.655 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T03:38:47.718 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T03:38:47.719 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:38:47.719 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T03:38:47.783 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T03:38:47.848 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T03:38:47.849 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:38:47.849 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T03:38:47.913 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T03:38:47.978 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T03:38:47.978 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:38:47.978 DEBUG:teuthology.orchestra.run.smithi138:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T03:38:48.042 DEBUG:teuthology.orchestra.run.smithi138:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T03:38:48.106 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T03:38:48.121 INFO:tasks.workunit:Pulling workunits from ref d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:38:48.121 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T03:38:48.121 INFO:tasks.workunit:timeout=3h 2024-09-15T03:38:48.121 INFO:tasks.workunit:cleanup=True 2024-09-15T03:38:48.121 DEBUG:teuthology.orchestra.run.smithi138:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:38:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T03:38:48.162 INFO:teuthology.orchestra.run.smithi138.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T03:38:48.162 DEBUG:teuthology.orchestra.run.smithi138:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T03:38:48.217 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T03:38:48.218 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T03:38:48.273 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout d75f23820806da622f561ee5ee7551015285993f 2024-09-15T03:38:48.328 INFO:tasks.workunit.client.0.smithi138.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T03:39:37.516 INFO:tasks.workunit.client.0.smithi138.stderr:Note: switching to 'd75f23820806da622f561ee5ee7551015285993f'. 2024-09-15T03:39:37.516 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.516 INFO:tasks.workunit.client.0.smithi138.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: git switch -c 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:Or undo this operation with: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: git switch - 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T03:39:37.518 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T03:39:37.518 INFO:tasks.workunit.client.0.smithi138.stderr:HEAD is now at d75f2382080 Merge branch 'wip-67762-quincy' of https://github.com/nbalacha/ceph into wip-bharath9-testing-2024-09-13-1444-quincy 2024-09-15T03:39:37.520 DEBUG:teuthology.orchestra.run.smithi138:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T03:39:37.576 DEBUG:teuthology.orchestra.run.smithi138:> set -ex 2024-09-15T03:39:37.576 DEBUG:teuthology.orchestra.run.smithi138:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T03:39:37.637 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-09-15T03:39:37.638 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-09-15T03:39:37.638 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-09-15T03:39:37.705 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/bad-inc-map 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-09-15T03:39:37.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-09-15T03:39:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:52997 2024-09-15T03:39:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:52997 2024-09-15T03:39:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-09-15T03:39:37.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-09-15T03:39:37.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none ' 2024-09-15T03:39:37.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:52997 ' 2024-09-15T03:39:37.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-09-15T03:39:37.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-09-15T03:39:37.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T03:39:37.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:39:37.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:39:37.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:39:37.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:39:37.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:39:37.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:39:37.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:39:37.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:39:37.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:39:37.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-15T03:39:37.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:39:37.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:37.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:37.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26755 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:37.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26755 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-09-15T03:39:37.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-09-15T03:39:37.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:39:37.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:39:37.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:39:37.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-09-15T03:39:37.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-09-15T03:39:38.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:39:38.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:39:38.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:39:38.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:39:38.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:39:38.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:39:38.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26755/ceph-mon.a.asok 2024-09-15T03:39:38.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:39:38.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26755/ceph-mon.a.asok config get fsid 2024-09-15T03:39:38.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:39:38.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:39:38.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:39:38.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:39:38.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:39:38.502 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:39:38.502 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26755/ceph-mon.a.asok 2024-09-15T03:39:38.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:39:38.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26755/ceph-mon.a.asok config get mon_host 2024-09-15T03:39:38.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-09-15T03:39:38.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-09-15T03:39:38.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:39:38.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:39:38.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:39:38.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-09-15T03:39:38.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:39:39.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:39:39.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:39:39.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 ' 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:39:39.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:39:39.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-09-15T03:39:39.439 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:39:39.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bdec87f1-9122-48fd-a84f-a5ea2fcf576c 2024-09-15T03:39:39.440 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 bdec87f1-9122-48fd-a84f-a5ea2fcf576c 2024-09-15T03:39:39.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bdec87f1-9122-48fd-a84f-a5ea2fcf576c' 2024-09-15T03:39:39.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:39:39.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7VuZmt+/RGhAA5o8B1bahJg+X4U7p2fZ8Fw== 2024-09-15T03:39:39.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7VuZmt+/RGhAA5o8B1bahJg+X4U7p2fZ8Fw=="}' 2024-09-15T03:39:39.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bdec87f1-9122-48fd-a84f-a5ea2fcf576c -i td/bad-inc-map/0/new.json 2024-09-15T03:39:39.782 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:39:39.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-09-15T03:39:39.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7VuZmt+/RGhAA5o8B1bahJg+X4U7p2fZ8Fw== --osd-uuid bdec87f1-9122-48fd-a84f-a5ea2fcf576c 2024-09-15T03:39:39.871 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.869+0000 7f070631e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:39.872 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.871+0000 7f070631e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:39.873 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.872+0000 7f070631e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:39.874 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.872+0000 7f070631e440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-15T03:39:39.874 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.872+0000 7f070631e440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-15T03:39:39.874 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.872+0000 7f070631e440 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-09-15T03:39:39.952 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:39.950+0000 7f070631e440 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-09-15T03:39:46.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-09-15T03:39:46.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:39:46.073 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:39:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:39:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:39:46.567 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:39:46.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:39:46.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:39:46.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:39:46.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:39:46.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:39:46.589 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:46.588+0000 7feea4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:46.615 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:46.615+0000 7feea4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:46.616 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:46.616+0000 7feea4e10440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:39:46.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:39:46.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:39:46.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:39:46.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:39:46.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:39:46.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:46.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:39:46.956 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:39:46.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:46.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:47.152 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:47.151+0000 7feea4e10440 -1 Falling back to public interface 2024-09-15T03:39:47.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:48.306 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:39:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:39:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:48.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:49.644 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:39:49.643+0000 7feea4e10440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:39:49.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:49.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:49.657 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:39:49.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:39:49.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:49.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:50.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:51.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:51.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:51.020 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:39:51.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:39:51.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:51.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:51.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:52.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:52.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:52.475 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:39:52.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:39:52.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:52.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:52.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:53.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:53.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:53.872 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:39:53.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:39:53.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:53.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:54.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:55.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:55.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:55.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:39:55.388 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:39:55.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:55.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:55.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:56.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:56.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:56.817 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:39:56.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:39:56.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:56.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:57.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:58.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:58.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:58.324 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:39:58.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:39:58.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:58.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:39:58.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:39:59.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:39:59.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:39:59.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:39:59.704 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:39:59.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:39:59.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:00.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:01.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:01.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:01.101 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:40:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:40:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:01.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:02.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:02.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:02.713 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:40:02.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:40:02.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:02.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:03.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:04.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:04.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:04.479 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:40:04.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:40:04.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:04.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:05.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:06.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:06.412 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:40:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:40:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:07.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:08.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:08.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:08.327 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:40:08.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:40:08.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:08.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:11.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:11.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:11.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:40:11.001 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:40:11.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:11.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:11.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:12.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:12.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:12.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:40:12.651 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:40:12.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:12.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:13.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:14.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:14.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:14.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:40:14.563 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:40:14.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:14.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:40:14.924 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27848,v1:127.0.0.1:6803/27848] [v2:127.0.0.1:6804/27848,v1:127.0.0.1:6805/27848] exists,up bdec87f1-9122-48fd-a84f-a5ea2fcf576c 2024-09-15T03:40:14.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:40:14.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 ' 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:40:14.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:40:14.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:40:14.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-09-15T03:40:14.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:40:14.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a97090f2-0594-4b0f-b9f1-68940dde1a81 2024-09-15T03:40:14.931 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 a97090f2-0594-4b0f-b9f1-68940dde1a81 2024-09-15T03:40:14.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a97090f2-0594-4b0f-b9f1-68940dde1a81' 2024-09-15T03:40:14.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:40:14.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeV+ZmaiP6NxAAHd68+4Z+apsk7kNxnIQ3AQ== 2024-09-15T03:40:14.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeV+ZmaiP6NxAAHd68+4Z+apsk7kNxnIQ3AQ=="}' 2024-09-15T03:40:14.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a97090f2-0594-4b0f-b9f1-68940dde1a81 -i td/bad-inc-map/1/new.json 2024-09-15T03:40:15.417 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:40:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-09-15T03:40:15.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeV+ZmaiP6NxAAHd68+4Z+apsk7kNxnIQ3AQ== --osd-uuid a97090f2-0594-4b0f-b9f1-68940dde1a81 2024-09-15T03:40:15.451 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.450+0000 7f6b7a529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:15.453 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.452+0000 7f6b7a529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:15.454 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.453+0000 7f6b7a529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:15.454 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.453+0000 7f6b7a529440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-15T03:40:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.453+0000 7f6b7a529440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-15T03:40:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.453+0000 7f6b7a529440 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-09-15T03:40:15.499 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:15.499+0000 7f6b7a529440 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-09-15T03:40:21.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-09-15T03:40:21.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:40:21.042 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T03:40:21.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:40:21.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:40:21.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:40:21.519 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:40:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:40:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:40:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:40:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:40:21.540 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:21.539+0000 7f5730b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:21.542 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:21.541+0000 7f5730b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:21.543 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:21.542+0000 7f5730b1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:21.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:40:21.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:40:21.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:40:21.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:40:21.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:40:21.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:21.865 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:40:21.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:40:21.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:21.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:22.060 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:22.059+0000 7f5730b1c440 -1 Falling back to public interface 2024-09-15T03:40:22.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:23.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:23.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:23.221 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:40:23.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:40:23.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:23.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:23.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:24.551 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:24.550+0000 7f5730b1c440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:40:24.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:24.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:24.589 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:40:24.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:40:24.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:24.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:25.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:26.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:26.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:26.019 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:40:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:40:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:26.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:27.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:27.505 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:40:27.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:40:27.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:27.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:28.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:28.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:28.951 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:40:28.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:40:28.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:28.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:29.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:30.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:30.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:30.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:40:30.333 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:40:30.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:30.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:30.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:31.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:31.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:31.696 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:40:31.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:40:31.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:31.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:32.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:33.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:33.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:33.088 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:40:33.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:40:33.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:33.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:33.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:34.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:34.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:34.448 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:40:34.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:40:34.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:34.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:34.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:35.849 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:40:35.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:40:35.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:35.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:36.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:37.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:37.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:37.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:40:37.254 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:40:37.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:37.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:38.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:39.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:39.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:39.067 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:40:39.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:40:39.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:39.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:39.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:40.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:40.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:40.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:40:40.842 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:40:40.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:40.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:41.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:42.774 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:40:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:40:42.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:42.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:44.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:45.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:45.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:45.018 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:40:45.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:40:45.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:45.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:45.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:46.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:46.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:46.756 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:40:46.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:40:46.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:46.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:47.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:48.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:48.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:48.363 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:40:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:40:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:40:48.826 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29658,v1:127.0.0.1:6811/29658] [v2:127.0.0.1:6812/29658,v1:127.0.0.1:6813/29658] exists,up a97090f2-0594-4b0f-b9f1-68940dde1a81 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-09-15T03:40:48.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 ' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:40:48.831 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:40:48.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52d627cf-ca72-427c-a020-7eb8094e1682 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 52d627cf-ca72-427c-a020-7eb8094e1682' 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 52d627cf-ca72-427c-a020-7eb8094e1682 2024-09-15T03:40:48.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:40:48.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAV+ZmAogvMhAAKkd5KRy03eUjljASurTBOg== 2024-09-15T03:40:48.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAV+ZmAogvMhAAKkd5KRy03eUjljASurTBOg=="}' 2024-09-15T03:40:48.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52d627cf-ca72-427c-a020-7eb8094e1682 -i td/bad-inc-map/2/new.json 2024-09-15T03:40:49.272 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:40:49.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-09-15T03:40:49.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAV+ZmAogvMhAAKkd5KRy03eUjljASurTBOg== --osd-uuid 52d627cf-ca72-427c-a020-7eb8094e1682 2024-09-15T03:40:49.306 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.305+0000 7f4c65b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:49.318 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.317+0000 7f4c65b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:49.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.318+0000 7f4c65b35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:49.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.318+0000 7f4c65b35440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-15T03:40:49.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.318+0000 7f4c65b35440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-15T03:40:49.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.318+0000 7f4c65b35440 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-09-15T03:40:49.349 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:49.349+0000 7f4c65b35440 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-09-15T03:40:55.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-09-15T03:40:55.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:40:55.216 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T03:40:55.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:40:55.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:40:55.702 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:40:55.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:40:55.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=16bee696-a273-47f0-a655-4292e19f7923 --auth-supported=none --mon-host=127.0.0.1:52997 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.26755/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:40:55.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:40:55.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:40:55.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:40:55.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:55.722+0000 7f504e984440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:55.759 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:55.758+0000 7f504e984440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:55.760 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:55.759+0000 7f504e984440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:40:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:40:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:40:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:40:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:40:56.063 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:40:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:40:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:40:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:40:56.274 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:56.273+0000 7f504e984440 -1 Falling back to public interface 2024-09-15T03:40:56.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:40:57.425 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:40:57.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:57.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:40:57.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:40:58.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:40:58.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:40:58.807 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:40:58.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:40:58.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:40:58.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:40:58.821 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:40:58.820+0000 7f504e984440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:40:59.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:00.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:00.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:00.278 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:41:00.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:41:00.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:00.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:00.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:01.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:01.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:01.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:41:01.770 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:41:01.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:01.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:02.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:41:03.156 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:41:03.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:03.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:03.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:04.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:04.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:04.584 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:41:04.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:41:04.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:04.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:04.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:05.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:05.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:05.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:41:05.980 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:41:05.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:05.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:06.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:07.394 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:41:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:41:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:07.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:08.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:08.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:08.911 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:41:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:41:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:09.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:10.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:10.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:10.299 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:41:10.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:41:10.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:10.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:11.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:12.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:12.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:12.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:41:12.325 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:41:12.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:12.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:13.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:14.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:14.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:14.204 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:41:14.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:41:14.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:14.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:14.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:41:15.901 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:41:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:17.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:17.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:17.584 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:41:17.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:41:17.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:17.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:18.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:19.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:19.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:19.356 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:41:19.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:41:19.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:19.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:20.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:21.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:21.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:21.036 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:41:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:41:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:21.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:21.451 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31432,v1:127.0.0.1:6819/31432] [v2:127.0.0.1:6820/31432,v1:127.0.0.1:6821/31432] exists,up 52d627cf-ca72-427c-a020-7eb8094e1682 2024-09-15T03:41:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:41:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:41:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:41:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-09-15T03:41:21.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-09-15T03:41:21.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-09-15T03:41:22.537 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T03:41:22.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:41:23.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-09-15T03:41:24.654 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 1 2024-09-15T03:41:24.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-09-15T03:41:25.939 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T03:41:25.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-09-15T03:41:30.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T03:41:30.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-09-15T03:41:30.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:41:30.952 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:41:30.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:30.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:31.303 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/27848,v1:127.0.0.1:6803/27848] [v2:127.0.0.1:6804/27848,v1:127.0.0.1:6805/27848] exists,up bdec87f1-9122-48fd-a84f-a5ea2fcf576c 2024-09-15T03:41:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:41:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:41:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:41:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:41:31.305 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:41:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:41:31.669 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29658,v1:127.0.0.1:6811/29658] [v2:127.0.0.1:6812/29658,v1:127.0.0.1:6813/29658] exists,up a97090f2-0594-4b0f-b9f1-68940dde1a81 2024-09-15T03:41:31.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:41:31.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:41:31.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:41:31.671 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:41:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:41:32.062 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/31432,v1:127.0.0.1:6819/31432] [v2:127.0.0.1:6820/31432,v1:127.0.0.1:6821/31432] exists,up 52d627cf-ca72-427c-a020-7eb8094e1682 2024-09-15T03:41:32.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:41:32.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:41:32.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:41:32.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:22.388+0000 7f503f068640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:22.909+0000 7f503f068640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:23.315+0000 7f503f068640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.326+0000 7f503d064640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.736+0000 7f503d064640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.064 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.737+0000 7f503d064640 -1 osd.2 21 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.056+0000 7f503f068640 -1 osd.2 22 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.559+0000 7f503d064640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.560+0000 7f503d064640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.936+0000 7f503f068640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:26.355+0000 7f503f068640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:26.357+0000 7f503f068640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-09-15T03:41:32.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-09-15T03:41:32.066 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:22.388+0000 7f503f068640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-09-15T03:41:32.066 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:22.910+0000 7f503f068640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-09-15T03:41:32.066 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:23.316+0000 7f503f068640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-09-15T03:41:32.066 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.326+0000 7f503d064640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.737+0000 7f503d064640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:24.737+0000 7f503d064640 10 osd.2 21 handle_osd_map bailing because last < start (21<22) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.057+0000 7f503f068640 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.560+0000 7f503d064640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.560+0000 7f503d064640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:25.936+0000 7f503f068640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:26.356+0000 7f503f068640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T03:41:26.357+0000 7f503f068640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-09-15T03:41:32.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-09-15T03:41:32.068 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T03:41:32.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-09-15T03:41:32.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-09-15T03:41:32.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T03:41:32.526 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' does not exist 2024-09-15T03:41:32.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-09-15T03:41:32.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:41:32.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:41:32.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:41:32.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:41:32.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:41:37.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:41:37.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-09-15T03:41:37.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-15T03:41:37.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:41:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:41:37.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:41:37.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:41:37.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:41:37.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:41:37.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:41:37.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:41:37.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:41:37.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:37.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:41:37.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:41:37.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:37.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:41:37.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:41:37.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-15T03:41:37.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:41:37.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:37.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:41:37.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26755 2024-09-15T03:41:37.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/bad-inc-map 0 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:41:37.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:41:37.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:41:37.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:41:37.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:41:37.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:41:37.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:41:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:41:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:41:37.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:41:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:41:37.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:41:37.887 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:37.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:41:37.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T03:41:37.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-09-15T03:41:37.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:41:37.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:37.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26755 2024-09-15T03:41:37.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26755 2024-09-15T03:41:37.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:41:37.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:41:37.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T03:41:37.895 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:41:37.895 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:41:37.966 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-09-15T03:41:37.967 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-09-15T03:41:38.031 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T03:41:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/divergent-priors 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-09-15T03:41:38.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-09-15T03:41:38.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-09-15T03:41:38.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-09-15T03:41:38.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-09-15T03:41:38.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-09-15T03:41:38.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-09-15T03:41:38.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-09-15T03:41:38.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-09-15T03:41:38.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-09-15T03:41:38.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-09-15T03:41:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-09-15T03:41:38.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none ' 2024-09-15T03:41:38.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-15T03:41:38.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T03:41:38.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-09-15T03:41:38.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:41:38.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-09-15T03:41:38.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_divergent_2 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_divergent_3 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_divergent_ec' 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-15T03:41:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:41:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:41:38.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:41:38.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:41:38.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:41:38.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:41:38.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:41:38.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:41:38.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:41:38.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:38.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:41:38.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:41:38.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:41:38.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:41:38.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:41:38.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:41:38.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:41:38.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:38.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:38.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:41:38.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:41:38.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:41:38.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-15T03:41:38.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:41:38.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:38.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:38.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32635 2024-09-15T03:41:38.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:41:38.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T03:41:38.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T03:41:38.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:41:38.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-09-15T03:41:38.061 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-09-15T03:41:38.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-15T03:41:38.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:41:38.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:41:38.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:41:38.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-15T03:41:38.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:41:38.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:41:38.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:41:38.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:41:38.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:41:38.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:41:38.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:41:38.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:41:38.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:41:38.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:41:38.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:41:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get fsid 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:41:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:41:38.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get mon_host 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:41:38.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-15T03:41:38.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:41:39.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:41:39.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:41:39.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:41:39.657 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:41:39.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:39.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:39.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:41:39.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:41:39.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:41:39.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T03:41:39.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-09-15T03:41:39.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:41:39.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:41:39.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:41:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 27e9679f-cb22-4905-86ae-0f6013fda3df' 2024-09-15T03:41:39.692 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:41:39.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:41:39.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzV+Zm+sU2KhAApVzsk/SnaIGC7i+XPQc36Q== 2024-09-15T03:41:39.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzV+Zm+sU2KhAApVzsk/SnaIGC7i+XPQc36Q=="}' 2024-09-15T03:41:39.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 27e9679f-cb22-4905-86ae-0f6013fda3df -i td/divergent-priors/0/new.json 2024-09-15T03:41:40.011 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:41:40.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T03:41:40.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzV+Zm+sU2KhAApVzsk/SnaIGC7i+XPQc36Q== --osd-uuid 27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:41:40.053 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.052+0000 7f95acc8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:40.057 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.056+0000 7f95acc8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:40.058 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.057+0000 7f95acc8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:40.058 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.057+0000 7f95acc8b440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:41:40.058 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.057+0000 7f95acc8b440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:41:40.059 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.057+0000 7f95acc8b440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:41:40.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:40.096+0000 7f95acc8b440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T03:41:45.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T03:41:45.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:41:45.490 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:41:45.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:41:45.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:41:45.983 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:41:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:41:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:41:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:41:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:41:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:41:46.014 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:46.013+0000 7ff45d71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:46.039 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:46.038+0000 7ff45d71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:46.040 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:46.039+0000 7ff45d71f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:41:46.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:41:46.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:41:46.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:41:46.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:41:46.380 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:41:46.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:41:46.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:46.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:41:46.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:46.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:46.575 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:46.574+0000 7ff45d71f440 -1 Falling back to public interface 2024-09-15T03:41:46.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:47.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:47.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:47.738 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:41:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:41:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:48.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:49.042 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:41:49.041+0000 7ff45d71f440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:41:49.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:49.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:49.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:41:49.102 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:41:49.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:49.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:49.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:50.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:50.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:50.629 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:41:50.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:41:50.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:50.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:51.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:52.041 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:41:52.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:41:52.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:52.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:52.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:53.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:53.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:53.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:41:53.429 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:41:53.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:53.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:53.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:54.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:54.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:54.837 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:41:54.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:41:54.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:54.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:56.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:56.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:56.203 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:41:56.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:41:56.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:56.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:56.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:57.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:57.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:57.719 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:41:57.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:41:57.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:57.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:58.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:41:59.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:41:59.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:41:59.129 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:41:59.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:41:59.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:41:59.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:41:59.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:00.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:00.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:00.526 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:42:00.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:42:00.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:00.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:01.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:02.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:02.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:02.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:42:02.464 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:42:02.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:02.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:03.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:04.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:04.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:04.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:42:04.029 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:42:04.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:04.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:04.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:42:05.735 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:42:05.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:05.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:06.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:07.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:07.857 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:42:07.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:42:07.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:07.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:08.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:09.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:09.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:09.290 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:42:09.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:42:09.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:09.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:10.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:11.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:11.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:11.207 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:42:11.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:42:11.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:11.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:42:11.600 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/33732,v1:127.0.0.1:6803/33732] [v2:127.0.0.1:6804/33732,v1:127.0.0.1:6805/33732] exists,up 27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:42:11.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:42:11.605 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:42:11.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:42:11.607 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:42:11.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:42:11.608 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:42:11.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2' 2024-09-15T03:42:11.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:42:11.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTV+ZmBUsVJRAAYYEu1AWS870AIj621hnLcA== 2024-09-15T03:42:11.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTV+ZmBUsVJRAAYYEu1AWS870AIj621hnLcA=="}' 2024-09-15T03:42:11.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 -i td/divergent-priors/1/new.json 2024-09-15T03:42:12.223 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:42:12.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T03:42:12.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTV+ZmBUsVJRAAYYEu1AWS870AIj621hnLcA== --osd-uuid d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:42:12.266 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.265+0000 7f81ac013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:12.281 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.280+0000 7f81ac013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:12.282 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.281+0000 7f81ac013440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:12.282 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.281+0000 7f81ac013440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:42:12.282 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.281+0000 7f81ac013440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:42:12.282 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.281+0000 7f81ac013440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:42:12.347 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:12.346+0000 7f81ac013440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T03:42:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T03:42:18.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:42:18.202 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T03:42:18.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:42:18.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:42:18.691 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:42:18.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:42:18.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:42:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:42:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:42:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:42:18.722 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:18.720+0000 7fb478410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:18.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:18.722+0000 7fb478410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:18.724 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:18.723+0000 7fb478410440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:19.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:42:19.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:42:19.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:42:19.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:42:19.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:42:19.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:19.055 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:42:19.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:42:19.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:19.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:19.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:19.238+0000 7fb478410440 -1 Falling back to public interface 2024-09-15T03:42:19.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:42:20.426 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:42:20.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:20.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:20.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:21.784 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:21.782+0000 7fb478410440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:42:21.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:21.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:21.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:42:21.831 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:42:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:22.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:23.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:23.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:23.295 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:42:23.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:42:23.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:23.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:23.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:24.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:24.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:24.759 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:42:24.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:42:24.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:24.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:25.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:26.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:26.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:26.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:42:26.164 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:42:26.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:26.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:26.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:27.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:27.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:27.631 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:42:27.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:42:27.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:27.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:28.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:29.144 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:42:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:42:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:29.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:30.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:30.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:30.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:42:30.609 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:42:30.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:30.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:31.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:32.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:32.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:32.031 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:42:32.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:42:32.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:32.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:32.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:33.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:33.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:33.432 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:42:33.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:42:33.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:33.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:33.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:34.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:34.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:34.807 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:42:34.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:42:34.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:34.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:35.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:36.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:36.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:36.456 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:42:36.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:42:36.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:36.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:37.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:38.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:38.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:38.118 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:42:38.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:42:38.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:38.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:38.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:39.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:39.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:39.861 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:42:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:42:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:40.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:41.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:41.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:41.664 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:42:41.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:42:41.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:41.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:42.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:43.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:43.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:43.375 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:42:43.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:42:43.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:43.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:44.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:45.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:45.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:45.110 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:42:45.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:42:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:42:45.646 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/35461,v1:127.0.0.1:6811/35461] [v2:127.0.0.1:6812/35461,v1:127.0.0.1:6813/35461] exists,up d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:42:45.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:42:45.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:42:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 61d51993-c1f3-48b9-abce-06f345cc874e' 2024-09-15T03:42:45.653 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:42:45.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:42:45.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1V+Zm58HbJxAAYPLv8pu8ieDWiwKqqbmL8Q== 2024-09-15T03:42:45.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1V+Zm58HbJxAAYPLv8pu8ieDWiwKqqbmL8Q=="}' 2024-09-15T03:42:45.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 61d51993-c1f3-48b9-abce-06f345cc874e -i td/divergent-priors/2/new.json 2024-09-15T03:42:46.291 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:42:46.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T03:42:46.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC1V+Zm58HbJxAAYPLv8pu8ieDWiwKqqbmL8Q== --osd-uuid 61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:42:46.334 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.333+0000 7f4ada809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:46.349 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.348+0000 7f4ada809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:46.350 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.349+0000 7f4ada809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:46.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.350+0000 7f4ada809440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:42:46.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.350+0000 7f4ada809440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:42:46.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.350+0000 7f4ada809440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:42:46.424 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:46.423+0000 7f4ada809440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T03:42:52.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T03:42:52.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:42:52.267 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T03:42:52.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:42:52.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:42:52.750 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:42:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:42:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:42:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:42:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:42:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:42:52.780 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:52.778+0000 7fd783a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:52.807 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:52.805+0000 7fd783a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:52.810 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:52.806+0000 7fd783a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:42:53.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:42:53.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:42:53.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:42:53.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:53.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:42:53.337 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:53.335+0000 7fd783a0b440 -1 Falling back to public interface 2024-09-15T03:42:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:54.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:54.557 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:42:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:42:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:42:54.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:55.905 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:42:55.903+0000 7fd783a0b440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:42:55.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:55.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:55.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:42:55.922 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:42:55.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:55.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:42:56.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:57.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:57.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:57.381 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:42:57.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:42:57.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:57.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:42:57.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:42:58.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:42:58.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:42:58.894 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:42:58.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:42:58.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:42:58.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:42:59.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:00.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:00.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:00.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:43:00.260 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:43:00.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:00.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:00.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:01.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:01.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:01.749 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:43:01.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:43:01.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:01.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:02.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:03.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:03.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:03.214 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:43:03.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:43:03.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:03.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:03.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:04.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:04.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:04.739 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:43:04.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:43:04.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:04.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:05.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:06.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:06.100 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:43:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:43:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:06.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:07.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:07.490 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:43:07.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:07.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:43:07.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:07.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:07.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:08.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:08.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:08.911 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:43:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:43:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:08.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:09.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:10.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:10.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:10.555 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:43:10.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:43:10.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:10.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:11.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:12.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:12.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:12.260 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:43:12.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:43:12.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:12.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:12.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:13.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:13.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:13.882 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:43:13.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:43:13.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:13.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:14.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:15.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:15.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:15.476 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:43:15.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:43:15.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:15.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:16.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:17.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:17.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:17.427 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:43:17.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:43:17.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:17.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:18.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:43:19.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:43:19.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:43:19.140 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:43:19.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:43:19.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:43:19.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:43:19.511 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/37215,v1:127.0.0.1:6819/37215] [v2:127.0.0.1:6820/37215,v1:127.0.0.1:6821/37215] exists,up 61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:43:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:43:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:43:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:43:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-09-15T03:43:20.345 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T03:43:20.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-09-15T03:43:21.014 INFO:tasks.workunit.client.0.smithi138.stderr:noin is set 2024-09-15T03:43:21.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-09-15T03:43:21.684 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T03:43:21.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-09-15T03:43:21.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T03:43:22.354 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T03:43:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:43:23.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-09-15T03:43:24.078 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T03:43:24.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-09-15T03:43:24.748 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T03:43:24.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-09-15T03:43:24.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:43:24.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:43:25.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:43:25.172 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:43:25.172 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:43:25.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:43:25.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:25.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:43:25.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-15T03:43:25.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-15T03:43:25.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-15T03:43:25.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:25.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:43:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-15T03:43:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-15T03:43:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972' 2024-09-15T03:43:25.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:25.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-15T03:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-15T03:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672972 2-64424509448' 2024-09-15T03:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-15T03:43:25.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:25.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:43:25.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-15T03:43:25.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:25.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-15T03:43:25.543 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836498 2024-09-15T03:43:25.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-15T03:43:25.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:25.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-15T03:43:25.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:43:26.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:43:26.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-15T03:43:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:43:28.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T03:43:28.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:28.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-15T03:43:28.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:28.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-15T03:43:28.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:28.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:43:28.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-15T03:43:28.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:28.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-15T03:43:28.628 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672972 2024-09-15T03:43:28.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-15T03:43:28.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:43:28.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672972 2024-09-15T03:43:28.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:28.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-15T03:43:28.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:43:28.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-15T03:43:28.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:28.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-15T03:43:28.985 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509448 2024-09-15T03:43:28.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T03:43:28.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:43:29.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T03:43:29.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-09-15T03:43:29.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:43:29.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:43:29.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:43:29.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:43:29.368 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:43:29.368 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:43:29.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:43:29.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:43:29.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:43:29.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:43:29.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:43:29.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:43:29.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:43:29.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:43:29.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:29.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:43:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836501 2024-09-15T03:43:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836501 2024-09-15T03:43:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501' 2024-09-15T03:43:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:29.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:43:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-15T03:43:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-15T03:43:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-09-15T03:43:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:30.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:43:30.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-15T03:43:30.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-15T03:43:30.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509451' 2024-09-15T03:43:30.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:30.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836501 2024-09-15T03:43:30.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:30.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:43:30.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836501 2024-09-15T03:43:30.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:30.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836501 2024-09-15T03:43:30.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-15T03:43:30.195 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836501 2024-09-15T03:43:30.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:30.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836499 -lt 21474836501 2024-09-15T03:43:30.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:43:31.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:43:31.550 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:31.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-15T03:43:31.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:31.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-15T03:43:31.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:31.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:43:31.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-15T03:43:31.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:31.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-15T03:43:31.912 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672975 2024-09-15T03:43:31.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T03:43:31.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:43:32.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T03:43:32.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:32.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-15T03:43:32.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:32.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:43:32.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-15T03:43:32.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:32.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-15T03:43:32.307 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509451 2024-09-15T03:43:32.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T03:43:32.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:43:32.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T03:43:32.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:43:32.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:43:32.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:43:33.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:43:33.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:43:33.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:43:33.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:43:33.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:43:33.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:43:33.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:43:34.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:43:34.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:43:34.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:43:34.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-09-15T03:43:34.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-09-15T03:43:34.352 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:43:34.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-09-15T03:43:34.364 INFO:tasks.workunit.client.0.smithi138.stdout:primary and soon to be divergent is 1 2024-09-15T03:43:34.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-09-15T03:43:34.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-09-15T03:43:34.700 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:43:34.700 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T03:43:23.174050+0000 0'0 28:11 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:07:51.724734+0000 0 0 2024-09-15T03:43:34.700 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:43:34.700 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:43:34.700 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:43:34.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-09-15T03:43:34.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T03:43:34.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-09-15T03:43:34.711 INFO:tasks.workunit.client.0.smithi138.stdout:writing initial objects 2024-09-15T03:43:34.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-09-15T03:43:34.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:34.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-09-15T03:43:34.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:34.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-09-15T03:43:35.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-09-15T03:43:35.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-09-15T03:43:35.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-09-15T03:43:35.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-09-15T03:43:35.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-09-15T03:43:35.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-09-15T03:43:35.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:35.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-09-15T03:43:36.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-09-15T03:43:36.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-09-15T03:43:36.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-09-15T03:43:36.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-09-15T03:43:36.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-09-15T03:43:36.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-09-15T03:43:36.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:36.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-09-15T03:43:37.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-09-15T03:43:37.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-09-15T03:43:37.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-09-15T03:43:37.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-09-15T03:43:37.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-09-15T03:43:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:37.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-09-15T03:43:38.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-09-15T03:43:38.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-09-15T03:43:38.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-09-15T03:43:38.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-09-15T03:43:38.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-09-15T03:43:38.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-09-15T03:43:38.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:38.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-09-15T03:43:39.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-09-15T03:43:39.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-09-15T03:43:39.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-09-15T03:43:39.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-09-15T03:43:39.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-09-15T03:43:39.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-09-15T03:43:39.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:39.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-09-15T03:43:40.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-09-15T03:43:40.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-09-15T03:43:40.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-09-15T03:43:40.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-09-15T03:43:40.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-09-15T03:43:40.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-09-15T03:43:40.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:40.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-09-15T03:43:41.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-09-15T03:43:41.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-09-15T03:43:41.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-09-15T03:43:41.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-09-15T03:43:41.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-09-15T03:43:41.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:41.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-09-15T03:43:42.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-09-15T03:43:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-09-15T03:43:42.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-09-15T03:43:42.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-09-15T03:43:42.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-09-15T03:43:42.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:42.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-09-15T03:43:43.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-09-15T03:43:43.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-09-15T03:43:43.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-09-15T03:43:43.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-09-15T03:43:43.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-09-15T03:43:43.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-09-15T03:43:43.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:43.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-09-15T03:43:44.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-09-15T03:43:44.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-09-15T03:43:44.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-09-15T03:43:44.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-09-15T03:43:44.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-09-15T03:43:44.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:44.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-09-15T03:43:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-09-15T03:43:45.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-09-15T03:43:45.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-09-15T03:43:45.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-09-15T03:43:45.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-09-15T03:43:45.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-09-15T03:43:45.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:45.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-09-15T03:43:46.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-09-15T03:43:46.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-09-15T03:43:46.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-09-15T03:43:46.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-09-15T03:43:46.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-09-15T03:43:46.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:46.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-09-15T03:43:47.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-09-15T03:43:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-09-15T03:43:47.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-09-15T03:43:47.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-09-15T03:43:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-09-15T03:43:47.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-09-15T03:43:47.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:47.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-09-15T03:43:48.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-09-15T03:43:48.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-09-15T03:43:48.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-09-15T03:43:48.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-09-15T03:43:48.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-09-15T03:43:48.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-09-15T03:43:48.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:48.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-09-15T03:43:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-09-15T03:43:49.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:49.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-09-15T03:43:49.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:49.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-09-15T03:43:49.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:49.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-09-15T03:43:49.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-09-15T03:43:49.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:43:49.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:43:49.831 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:43:49.831 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:43:49.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:43:49.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:43:49.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:43:49.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:43:49.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:43:49.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:43:49.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:43:49.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:43:49.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:43:50.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:43:50.239 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:43:50.239 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:43:50.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:43:50.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:50.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:43:50.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836507 2024-09-15T03:43:50.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836507 2024-09-15T03:43:50.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507' 2024-09-15T03:43:50.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:50.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:43:50.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-15T03:43:50.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-15T03:43:50.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672981' 2024-09-15T03:43:50.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:43:50.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:43:50.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-15T03:43:50.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-15T03:43:50.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836507 1-42949672981 2-64424509457' 2024-09-15T03:43:50.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:50.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836507 2024-09-15T03:43:50.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:50.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:43:50.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836507 2024-09-15T03:43:50.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:50.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836507 2024-09-15T03:43:50.581 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836507 2024-09-15T03:43:50.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-15T03:43:50.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:50.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-15T03:43:50.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:43:51.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:43:51.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:43:52.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836507 2024-09-15T03:43:52.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:52.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-15T03:43:52.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:52.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:43:52.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-15T03:43:52.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:52.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-15T03:43:52.305 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672981 2024-09-15T03:43:52.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T03:43:52.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:43:52.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672981 2024-09-15T03:43:52.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:43:52.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-15T03:43:52.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:43:52.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:43:52.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-15T03:43:52.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:43:52.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-15T03:43:52.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T03:43:52.674 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509457 2024-09-15T03:43:52.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:43:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T03:43:53.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:43:53.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:43:53.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:43:53.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:43:53.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:43:53.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:43:53.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:43:53.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:43:53.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:43:53.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:43:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:43:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:43:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:43:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-09-15T03:43:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T03:43:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-15T03:43:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-15T03:43:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T03:43:54.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-09-15T03:43:54.769 INFO:tasks.workunit.client.0.smithi138.stdout:blackholing osds 0 2 2024-09-15T03:43:54.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-09-15T03:43:54.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-09-15T03:43:55.104 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:43:55.104 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-15T03:43:23.174050+0000 29'100 29:111 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:07:51.724734+0000 0 0 2024-09-15T03:43:55.104 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:43:55.104 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:43:55.105 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:43:55.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-15T03:43:55.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-09-15T03:43:55.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:43:55.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:43:55.116 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:43:55.116 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:43:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:43:55.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:43:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-15T03:43:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T03:43:55.188 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:43:55.188 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:43:55.188 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:43:55.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-09-15T03:43:55.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-09-15T03:43:55.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:43:55.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-09-15T03:43:55.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T03:43:55.267 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:43:55.267 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:43:55.268 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:43:55.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-09-15T03:43:55.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-09-15T03:43:55.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-09-15T03:43:55.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-09-15T03:43:55.276 INFO:tasks.workunit.client.0.smithi138.stdout:writing divergent object 2024-09-15T03:43:55.276 INFO:tasks.workunit.client.0.smithi138.stdout:create missing divergent object 2024-09-15T03:43:55.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-09-15T03:43:55.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-09-15T03:43:55.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=existing_99 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/divergent-priors 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T03:43:55.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test existing_99 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-09-15T03:43:55.280 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:0 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('1' '0' '2') 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T03:43:55.638 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/divergent-priors/1/type 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:43:55.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:43:55.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:43:55.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:43:55.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:43:55.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:43:55.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.32635/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T03:43:55.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T03:43:55.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T03:43:55.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T03:43:55.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T03:43:55.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:43:55.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:43:55.722 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:43:55.723 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:43:55.723 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:43:55.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:43:55.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T03:43:55.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok injectdataerr test existing_99 2024-09-15T03:43:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-09-15T03:43:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-09-15T03:44:05.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-09-15T03:44:05.826 INFO:tasks.workunit.client.0.smithi138.stderr:rados: no process found 2024-09-15T03:44:05.827 INFO:tasks.workunit.client.0.smithi138.stdout:killing all the osds 2024-09-15T03:44:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-09-15T03:44:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-09-15T03:44:06.167 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:44:06.167 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 active+clean 2024-09-15T03:43:56.000142+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:07:51.724734+0000 0 0 2024-09-15T03:44:06.167 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:44:06.167 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:44:06.167 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:44:06.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-09-15T03:44:06.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:44:06.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:44:06.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:44:06.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:44:06.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:44:06.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:44:06.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T03:44:06.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-09-15T03:44:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T03:44:06.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T03:44:06.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-09-15T03:44:07.531 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T03:44:07.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-09-15T03:44:07.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-09-15T03:44:08.047 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T03:44:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-09-15T03:44:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T03:44:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-09-15T03:44:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T03:44:09.054 INFO:tasks.workunit.client.0.smithi138.stdout:bringing up non_divergent 0 2 2024-09-15T03:44:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-09-15T03:44:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-09-15T03:44:09.433 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:44:09.433 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 48300 0 0 17 84 17 stale+active+clean 2024-09-15T03:43:56.000142+0000 29'101 29:120 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:07:51.724734+0000 0 0 2024-09-15T03:44:09.433 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:44:09.433 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:44:09.433 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:44:09.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:09.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:44:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:44:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:44:09.451 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:44:09.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:44:09.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T03:44:09.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:44:09.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:44:09.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:44:09.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:44:09.476 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:09.475+0000 7fb0b5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:09.477 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:09.477+0000 7fb0b5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:09.478 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:09.478+0000 7fb0b5e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:09.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:44:09.805 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:44:09.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:09.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:44:09.992 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:09.991+0000 7fb0b5e09440 -1 Falling back to public interface 2024-09-15T03:44:10.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:11.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:11.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:11.159 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:44:11.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:44:11.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:11.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:44:11.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:12.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:12.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:12.533 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:44:12.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:44:12.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:12.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:44:12.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:12.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:12.953+0000 7fb0b5e09440 -1 osd.0 29 log_to_monitors true 2024-09-15T03:44:13.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:13.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:13.899 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:44:13.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:44:13.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:13.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:44:14.256 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/41516,v1:127.0.0.1:6803/41516] [v2:127.0.0.1:6804/41516,v1:127.0.0.1:6805/41516] exists,up 27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:44:14.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:44:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:44:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:44:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-09-15T03:44:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:44:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:44:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:44:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:44:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:44:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:44:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:44:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:44:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:44:14.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:44:14.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:44:14.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:44:14.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:44:14.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:44:14.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:44:14.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:44:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:44:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:44:14.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:44:14.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:44:14.272 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:44:14.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:44:14.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:44:14.288 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:14.287+0000 7fabb731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:14.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:14.300+0000 7fabb731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:14.302 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:14.301+0000 7fabb731f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:44:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:44:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:44:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:44:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:44:14.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:14.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:44:14.619 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:44:14.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:14.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:44:14.816 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:14.815+0000 7fabb731f440 -1 Falling back to public interface 2024-09-15T03:44:14.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:15.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:15.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:15.977 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:44:15.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:44:15.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:15.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:44:16.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:17.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:17.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:17.336 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:44:17.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:44:17.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:17.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:44:17.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:44:17.784 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:17.783+0000 7fabb731f440 -1 osd.2 29 log_to_monitors true 2024-09-15T03:44:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:44:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:44:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:44:18.693 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:44:18.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:44:18.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:44:19.063 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/42095,v1:127.0.0.1:6811/42095] [v2:127.0.0.1:6812/42095,v1:127.0.0.1:6813/42095] exists,up 61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:44:19.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:44:19.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:44:19.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:44:19.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:19.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-09-15T03:44:19.669 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T03:44:19.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:19.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-09-15T03:44:20.460 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already in. 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:44:20.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:44:20.472 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:44:20.472 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:44:20.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:44:20.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:44:20.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:44:20.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:44:20.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:44:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:44:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:44:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:44:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:44:20.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:44:21.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822660 2024-09-15T03:44:21.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822660 2024-09-15T03:44:21.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660' 2024-09-15T03:44:21.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:44:21.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:44:21.153 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:44:21.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:44:21.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:44:21.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:44:21.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:44:21.155 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:44:21.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757251 2024-09-15T03:44:21.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757251 2024-09-15T03:44:21.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-15T03:44:21.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:44:21.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822660 2024-09-15T03:44:21.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:44:21.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:44:21.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822660 2024-09-15T03:44:21.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:44:21.290 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 154618822660 2024-09-15T03:44:21.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822660 2024-09-15T03:44:21.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-15T03:44:21.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:44:21.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-15T03:44:21.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:44:21.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757251 2024-09-15T03:44:21.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:44:21.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:44:21.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757251 2024-09-15T03:44:21.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:44:21.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757251 2024-09-15T03:44:21.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-15T03:44:21.679 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 163208757251 2024-09-15T03:44:21.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:44:22.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757249 -lt 163208757251 2024-09-15T03:44:22.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:44:23.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:44:23.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:44:23.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-15T03:44:23.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:44:23.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:23.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:23.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:44:23.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:23.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:23.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:23.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:23.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:23.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:23.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:24.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:24.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:24.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:44:24.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:44:24.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:44:24.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:24.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:24.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:24.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:24.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:25.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:25.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:25.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:25.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:25.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:25.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:26.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:26.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:26.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:44:26.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:26.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:44:26.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:26.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:26.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:26.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:26.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:26.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:26.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:26.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:26.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:26.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:26.721 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:26.721 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:27.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:27.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:27.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:27.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:27.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:44:27.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:27.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:28.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:28.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:28.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:28.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:28.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:28.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:28.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:28.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:28.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:28.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:29.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:44:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:29.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:30.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:30.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:30.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:30.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:30.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:31.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:31.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:31.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:31.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:44:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:44:33.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:33.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:33.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:33.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:33.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:33.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:33.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:34.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:34.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:34.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:34.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:34.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:44:34.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:44:37.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:37.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:37.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:37.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:38.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:38.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:38.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:38.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:38.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:38.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:44:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:45.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:45.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:45.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:45.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:45.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:46.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:46.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:46.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:46.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:46.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:46.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:44:46.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:44:46.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:44:54.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:44:54.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:44:54.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:44:54.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:44:54.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:44:54.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:44:54.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:44:55.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:44:55.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:44:55.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:44:55.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:44:55.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:44:55.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "f6f8d098-6673-4ccc-a624-3f02dc001a4e", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:44:55.953620+0000", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:44:55.955 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:44:55.956 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:44:55.957 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:44:55.958 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:44:55.959 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:41:38.089410Z", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:41:38.089410Z", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:44:55.960 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.961 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.962 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:44:55.963 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 42, 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:41:38.909648+0000", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:44:20.534327+0000", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:44:18.068190+0000", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:44:19.965416+0000", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:44:55.964 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:44:55.965 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:43:22.075253+0000", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T03:44:55.966 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:44:55.967 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "29", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T03:44:55.968 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:44:55.969 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:44:55.970 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "27e9679f-cb22-4905-86ae-0f6013fda3df", 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:44:55.971 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 36, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 40, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 30, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.972 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.973 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.974 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.975 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 41516 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/41516", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/41516", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/41516", 2024-09-15T03:44:55.976 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/41516", 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2", 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:44:55.977 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 23, 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 31, 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:44:55.978 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:44:55.979 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.980 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 35461 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/35461", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/35461", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/35461", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/35461", 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:44:55.982 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "61d51993-c1f3-48b9-abce-06f345cc874e", 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 38, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 0, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 32, 2024-09-15T03:44:55.983 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:44:55.984 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.985 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:44:55.986 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 42095 2024-09-15T03:44:55.987 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/42095", 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/42095", 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/42095", 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/42095", 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.988 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:44:06.588587+0000", 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:42:10.940788+0000", 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:44:55.989 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:44:07.145756+0000", 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:42:44.925366+0000", 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:44:07.856922+0000", 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:44:55.990 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:43:18.927875+0000", 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:44:55.991 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:44:55.992 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/41516,v1:127.0.0.1:6805/41516]", 2024-09-15T03:44:55.993 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:44:55.994 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:41:44.852529Z", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:44:55.995 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/41516,v1:127.0.0.1:6803/41516]", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/41516,v1:127.0.0.1:6809/41516]", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/41516,v1:127.0.0.1:6807/41516]", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:44:55.996 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:55.997 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/35461,v1:127.0.0.1:6813/35461]", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:44:55.998 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:42:17.551008Z", 2024-09-15T03:44:55.999 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/35461,v1:127.0.0.1:6811/35461]", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/35461,v1:127.0.0.1:6817/35461]", 2024-09-15T03:44:56.000 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/35461,v1:127.0.0.1:6815/35461]", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:44:56.001 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/42095,v1:127.0.0.1:6813/42095]", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:44:56.002 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:42:51.612151Z", 2024-09-15T03:44:56.003 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/42095,v1:127.0.0.1:6811/42095]", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/42095,v1:127.0.0.1:6817/42095]", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/42095,v1:127.0.0.1:6815/42095]", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:44:56.004 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.005 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [] 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 36 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 38 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.006 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 42, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:44:56.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:44:56.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.009 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.010 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:44:56.011 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.012 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:44:56.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:44:56.014 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:44:56.015 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:44:56.016 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:44:56.017 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:44:56.018 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:44:56.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:44:56.020 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:44:56.021 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:44:56.022 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:44:56.023 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.024 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 1, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:44:56.025 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48300, 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stderr:report 1703406081 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:44:56.026 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 100, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 100, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:44:56.027 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:44:56.028 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:44:56.029 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:44:56.030 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 16, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 16, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:44:56.031 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894716, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2124, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892533, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678084, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206358016, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:44:56.032 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2174976, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 423889, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 913954266 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:44:56.033 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 1, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:44:56.034 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:44:56.035 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894716, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2124, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892533, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678084, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206358016, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2174976, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 423889, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:44:56.036 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 913954266 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 1, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:44:56.037 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48300, 2024-09-15T03:44:56.038 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 100, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 100, 2024-09-15T03:44:56.039 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:44:56.040 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:44:56.041 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1228800, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 144900, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 16, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 16, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:44:56.042 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 154618822667 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672984 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 163208757258 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.043 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:44:56.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 5, 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:44:09.179546+0000", 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:44:56.046 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 513, 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:44:56.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-09-15T03:44:56.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-09-15T03:44:56.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-09-15T03:44:56.048 INFO:tasks.workunit.client.0.smithi138.stdout:writing non-divergent object existing_4 2024-09-15T03:44:56.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-09-15T03:44:56.307 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:44:56.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 29'100 42:123 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:44:56.307 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:44:56.307 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:44:56.308 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:44:56.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T03:44:56.425 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery 2024-09-15T03:44:56.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-09-15T03:44:56.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-09-15T03:44:56.759 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:44:56.759 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 29'100 42:123 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:44:56.759 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:44:56.759 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:44:56.760 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:44:56.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:56.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-15T03:44:56.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T03:44:56.847 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:44:56.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-09-15T03:44:56.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T03:44:56.931 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:44:56.938 INFO:tasks.workunit.client.0.smithi138.stdout:revive divergent 1 2024-09-15T03:44:56.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-09-15T03:44:56.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-09-15T03:44:57.272 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:44:57.272 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 29'100 42:123 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:44:57.272 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:44:57.272 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:44:57.273 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:44:57.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-09-15T03:44:57.943 INFO:tasks.workunit.client.0.smithi138.stderr:noup is set 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:44:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:44:57.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:44:57.959 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:44:57.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:44:57.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:44:57.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:44:57.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:44:57.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:44:57.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:44:57.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:44:57.986 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:57.986+0000 7f356b630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:57.996 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:57.995+0000 7f356b630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:57.997 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:57.996+0000 7f356b630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:44:58.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-15T03:44:58.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-09-15T03:44:58.515 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:44:58.515+0000 7f356b630440 -1 Falling back to public interface 2024-09-15T03:45:01.419 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:01.418+0000 7f356b630440 -1 osd.1 29 log_to_monitors true 2024-09-15T03:45:03.401 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery divergent 2024-09-15T03:45:03.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-09-15T03:45:03.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:03.747 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:03.747 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 42'101 43:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:45:03.747 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:03.748 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:03.748 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:03.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-09-15T03:45:03.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:45:03.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:45:03.758 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:45:03.759 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:03.759 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:03.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:45:03.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-09-15T03:45:03.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T03:45:03.835 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:45:03.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-09-15T03:45:04.691 INFO:tasks.workunit.client.0.smithi138.stderr:noup is unset 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:45:04.703 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:04.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:04.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:05.237 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/41516,v1:127.0.0.1:6803/41516] [v2:127.0.0.1:6804/41516,v1:127.0.0.1:6805/41516] exists,up 27e9679f-cb22-4905-86ae-0f6013fda3df 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:45:05.240 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:05.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:05.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:45:05.707 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/43748,v1:127.0.0.1:6819/43748] [v2:127.0.0.1:6820/43748,v1:127.0.0.1:6821/43748] exists,up d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:05.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:45:05.710 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:05.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:05.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:45:06.062 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/42095,v1:127.0.0.1:6811/42095] [v2:127.0.0.1:6812/42095,v1:127.0.0.1:6813/42095] exists,up 61d51993-c1f3-48b9-abce-06f345cc874e 2024-09-15T03:45:06.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:45:06.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:45:06.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:45:06.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:06.401 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:06.402 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 42'101 43:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:45:06.402 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:06.402 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:06.402 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:06.412 INFO:tasks.workunit.client.0.smithi138.stdout:wait for peering 2024-09-15T03:45:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-09-15T03:45:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:06.748 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:06.748 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:44:20.886730+0000 42'101 43:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:47:03.661036+0000 0 0 2024-09-15T03:45:06.748 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:06.748 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:06.749 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:06.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-09-15T03:45:06.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-09-15T03:45:06.917 INFO:tasks.workunit.client.0.smithi138.stdout:killing divergent 1 2024-09-15T03:45:06.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:07.254 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:07.254 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 2 2 0 0 47971 0 0 17 84 17 active+recovering+degraded 2024-09-15T03:45:05.312681+0000 42'101 49:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:56:37.225269+0000 0 0 2024-09-15T03:45:07.254 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:07.254 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:07.255 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:07.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T03:45:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:45:07.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:45:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:45:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:45:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:45:07.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:45:07.370 INFO:tasks.workunit.client.0.smithi138.stdout:reviving divergent 1 2024-09-15T03:45:07.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-09-15T03:45:07.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:07.704 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:07.704 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 2 2 0 0 47971 0 0 17 84 17 active+recovering+degraded 2024-09-15T03:45:05.312681+0000 42'101 49:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:56:37.225269+0000 0 0 2024-09-15T03:45:07.704 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:07.704 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:07.705 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:45:07.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:07.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:45:07.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:45:07.721 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:45:07.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:45:07.748 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:07.746+0000 7f52a491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:07.749 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:07.748+0000 7f52a491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:07.749 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:07.749+0000 7f52a491d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:08.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:45:08.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:45:08.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:08.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:45:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:08.266+0000 7f52a491d440 -1 Falling back to public interface 2024-09-15T03:45:08.435 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/43748,v1:127.0.0.1:6819/43748] [v2:127.0.0.1:6820/43748,v1:127.0.0.1:6821/43748] exists,up d0a0a9c0-3776-4e24-ac27-b3c7e745d0c2 2024-09-15T03:45:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:45:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:45:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:45:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-09-15T03:45:11.188 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:11.187+0000 7f52a491d440 -1 osd.1 49 log_to_monitors true 2024-09-15T03:45:28.437 INFO:tasks.workunit.client.0.smithi138.stdout:allowing recovery 2024-09-15T03:45:28.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-09-15T03:45:28.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:28.784 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:28.785 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-15T03:45:12.680403+0000 49'102 53:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:07:29.401145+0000 0 0 2024-09-15T03:45:28.785 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:28.785 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:28.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T03:45:28.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T03:45:28.900 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:45:28.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T03:45:28.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T03:45:29.014 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:45:29.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-09-15T03:45:29.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T03:45:29.128 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:45:29.136 INFO:tasks.workunit.client.0.smithi138.stdout:reading divergent objects 2024-09-15T03:45:29.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-09-15T03:45:29.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-09-15T03:45:29.473 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:45:29.473 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48454 0 0 18 84 18 active+clean 2024-09-15T03:45:12.680403+0000 49'102 53:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:43:22.125441+0000 0'0 2024-09-15T03:43:22.125441+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:07:29.401145+0000 0 0 2024-09-15T03:45:29.473 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:45:29.473 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:45:29.474 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:45:29.484 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-09-15T03:45:29.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-09-15T03:45:29.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:45:29.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T03:45:29.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:45:29.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T03:45:29.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:45:29.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T03:45:29.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:45:29.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T03:45:29.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-09-15T03:45:29.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-15T03:45:29.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:45:29.627 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-15T03:43:55.305461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:45:29.627 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-15T03:43:55.305461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:45:29.627 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4624.0:1 2024-09-15T03:43:55.305461+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:45:05.143+0000 7f354ee63640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-15T03:45:29.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-09-15T03:45:29.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:45:29.633 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T03:45:29.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-09-15T03:45:29.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-09-15T03:45:29.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T03:45:29.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T03:45:30.208 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T03:45:30.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-09-15T03:45:30.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:45:30.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:45:30.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:45:30.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:45:30.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:45:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:45:30.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:45:30.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:45:30.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:45:30.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:45:30.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:45:30.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:45:30.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:45:30.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:45:30.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:45:30.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:45:30.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:45:30.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:45:30.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:45:30.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:45:30.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:45:30.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:30.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:30.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:45:30.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:45:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:45:30.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:45:30.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:45:30.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:45:30.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:45:30.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:45:30.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:45:30.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:45:30.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:45:30.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:45:30.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:45:30.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:45:30.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:45:30.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:45:30.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:45:30.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:45:30.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:45:30.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:30.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:30.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:45:30.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:45:30.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:45:30.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-15T03:45:30.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:45:30.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:30.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32635 2024-09-15T03:45:30.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-09-15T03:45:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:45:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-15T03:45:30.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:45:31.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:45:31.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:45:31.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:45:31.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:45:31.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:45:31.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:45:31.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get fsid 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:45:31.263 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:31.264 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:31.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:45:31.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:45:31.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get mon_host 2024-09-15T03:45:31.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-15T03:45:31.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:45:32.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:45:32.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:45:32.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:45:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:45:32.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:45:32.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:45:32.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:45:32.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:45:32.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:45:32.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0433c468-fffb-4374-a487-5905fddfa18d' 2024-09-15T03:45:32.226 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:45:32.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:45:32.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcWOZmnNZyDhAAWTOCLFip5rgiSGh4JRmvpw== 2024-09-15T03:45:32.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcWOZmnNZyDhAAWTOCLFip5rgiSGh4JRmvpw=="}' 2024-09-15T03:45:32.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0433c468-fffb-4374-a487-5905fddfa18d -i td/divergent-priors/0/new.json 2024-09-15T03:45:32.525 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T03:45:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcWOZmnNZyDhAAWTOCLFip5rgiSGh4JRmvpw== --osd-uuid 0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:45:32.568 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.567+0000 7f4475014440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:32.570 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.569+0000 7f4475014440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:32.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.570+0000 7f4475014440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:32.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.571+0000 7f4475014440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:45:32.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.571+0000 7f4475014440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:45:32.572 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.571+0000 7f4475014440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:45:32.603 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:32.602+0000 7f4475014440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T03:45:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T03:45:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:45:37.886 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:45:37.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:45:37.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:45:38.379 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:45:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:45:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:45:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:45:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:45:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:45:38.410 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:38.409+0000 7f6ac080e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:38.412 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:38.411+0000 7f6ac080e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:38.413 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:38.412+0000 7f6ac080e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:45:38.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:45:38.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:45:38.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:45:38.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:45:38.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:45:38.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:38.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:45:38.740 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:45:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:38.938 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:38.938+0000 7f6ac080e440 -1 Falling back to public interface 2024-09-15T03:45:39.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:40.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:40.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:40.138 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:45:40.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:45:40.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:40.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:40.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:41.427 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:45:41.426+0000 7f6ac080e440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:45:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:45:41.499 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:45:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:41.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:42.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:42.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:42.997 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:45:42.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:45:42.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:42.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:43.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:45:44.393 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:45:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:44.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:45.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:45.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:45.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:45:45.825 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:45:45.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:45.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:46.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:47.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:47.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:47.260 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:45:47.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:45:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:47.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:48.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:48.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:48.655 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:45:48.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:45:48.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:48.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:49.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:50.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:50.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:50.146 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:45:50.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:45:50.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:50.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:50.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:51.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:51.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:51.535 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:45:51.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:45:51.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:51.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:51.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:52.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:52.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:52.902 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:45:52.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:45:52.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:52.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:53.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:54.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:54.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:54.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:45:54.830 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:45:54.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:54.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:55.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:56.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:56.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:56.395 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:45:56.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:45:56.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:56.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:57.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:58.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:58.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:58.181 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:45:58.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:45:58.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:58.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:45:58.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:45:59.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:45:59.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:45:59.806 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:45:59.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:45:59.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:45:59.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:46:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:01.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:01.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:46:01.413 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:46:01.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:01.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:46:02.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:03.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:03.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:03.117 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:46:03.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:46:03.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:03.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:46:03.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:04.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:04.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:04.639 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:46:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:46:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:46:05.008 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/46325,v1:127.0.0.1:6803/46325] [v2:127.0.0.1:6804/46325,v1:127.0.0.1:6805/46325] exists,up 0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:46:05.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:46:05.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:46:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:46:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T03:46:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-09-15T03:46:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:46:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:46:05.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:46:05.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:46:05.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:46:05.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:46:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:46:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:46:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:46:05.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:46:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:46:05.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:46:05.021 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:46:05.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:46:05.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c06d852a-9fbb-4f83-aa04-2a1803ed93d0' 2024-09-15T03:46:05.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:46:05.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9WOZmYjvKARAAddi/h0gvgsPOhX/arPBBNw== 2024-09-15T03:46:05.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9WOZmYjvKARAAddi/h0gvgsPOhX/arPBBNw=="}' 2024-09-15T03:46:05.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c06d852a-9fbb-4f83-aa04-2a1803ed93d0 -i td/divergent-priors/1/new.json 2024-09-15T03:46:05.604 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:46:05.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T03:46:05.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9WOZmYjvKARAAddi/h0gvgsPOhX/arPBBNw== --osd-uuid c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:46:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.646+0000 7fc5ac409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:05.649 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.648+0000 7fc5ac409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:05.650 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.649+0000 7fc5ac409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:05.651 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.649+0000 7fc5ac409440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:46:05.651 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.649+0000 7fc5ac409440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:46:05.651 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.649+0000 7fc5ac409440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:46:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:05.719+0000 7fc5ac409440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T03:46:11.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T03:46:11.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:46:11.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:46:11.669 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T03:46:11.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:46:12.159 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:46:12.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:46:12.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:46:12.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:46:12.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:46:12.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:46:12.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:12.188+0000 7f98b181e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:12.223 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:12.222+0000 7f98b181e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:12.224 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:12.224+0000 7f98b181e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:46:12.552 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:46:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:12.760+0000 7f98b181e440 -1 Falling back to public interface 2024-09-15T03:46:12.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:13.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:13.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:13.910 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:46:13.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:46:13.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:13.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:14.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:15.264 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:15.263+0000 7f98b181e440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:46:15.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:15.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:15.277 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:46:15.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:46:15.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:15.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:15.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:16.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:16.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:16.776 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:46:16.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:46:16.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:16.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:17.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:18.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:18.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:18.255 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:46:18.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:46:18.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:18.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:18.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:19.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:19.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:19.678 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:46:19.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:46:19.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:19.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:20.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:21.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:21.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:21.129 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:46:21.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:46:21.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:21.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:21.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:22.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:22.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:22.496 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:46:22.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:46:22.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:22.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:23.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:24.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:24.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:24.017 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:46:24.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:46:24.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:24.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:25.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:25.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:25.598 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:46:25.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:46:25.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:25.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:25.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:26.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:26.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:26.965 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:46:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:46:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:27.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:28.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:28.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:28.440 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:46:28.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:46:28.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:28.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:28.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:29.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:29.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:29.959 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:46:29.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:46:29.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:29.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:31.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:31.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:31.607 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:46:31.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:46:31.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:31.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:32.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:33.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:33.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:33.204 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:46:33.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:46:33.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:33.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:33.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:34.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:34.938 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:46:34.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:46:34.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:34.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:36.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:37.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:37.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:37.107 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:46:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:46:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:37.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:38.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:38.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:38.513 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:46:38.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:46:38.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:38.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:39.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:40.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:40.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:40.415 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T03:46:40.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:46:40.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:40.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:46:40.786 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/48081,v1:127.0.0.1:6811/48081] [v2:127.0.0.1:6812/48081,v1:127.0.0.1:6813/48081] exists,up c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-09-15T03:46:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:46:40.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:46:40.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9081311c-9dfa-4576-811b-7ba7a9f09c12' 2024-09-15T03:46:40.793 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:46:40.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:46:40.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCgWOZmsTw0MBAA1bQbg+9VysQKVsYMDE50Pg== 2024-09-15T03:46:40.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCgWOZmsTw0MBAA1bQbg+9VysQKVsYMDE50Pg=="}' 2024-09-15T03:46:40.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9081311c-9dfa-4576-811b-7ba7a9f09c12 -i td/divergent-priors/2/new.json 2024-09-15T03:46:41.282 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:46:41.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T03:46:41.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgWOZmsTw0MBAA1bQbg+9VysQKVsYMDE50Pg== --osd-uuid 9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:46:41.326 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.325+0000 7f21b8f29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:41.327 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.327+0000 7f21b8f29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:41.328 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.328+0000 7f21b8f29440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:41.329 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.328+0000 7f21b8f29440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:46:41.329 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.328+0000 7f21b8f29440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:46:41.329 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.328+0000 7f21b8f29440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:46:41.406 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:41.406+0000 7f21b8f29440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T03:46:47.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T03:46:47.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:46:47.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:46:47.632 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T03:46:47.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:46:48.144 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:46:48.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:46:48.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:46:48.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:46:48.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:46:48.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:46:48.175 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:48.174+0000 7f7bd300a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:48.202 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:48.201+0000 7f7bd300a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:48.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:48.202+0000 7f7bd300a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:48.583 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:46:48.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:46:48.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:48.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:48.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:48.738+0000 7f7bd300a440 -1 Falling back to public interface 2024-09-15T03:46:48.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:49.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:49.942 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:46:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:46:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:50.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:51.229 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:46:51.228+0000 7f7bd300a440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:46:51.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:51.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:51.304 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:46:51.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:46:51.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:51.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:51.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:52.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:52.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:52.965 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:46:52.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:46:52.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:52.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:53.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:54.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:54.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:54.364 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:46:54.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:46:54.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:54.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:54.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:55.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:55.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:55.763 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:46:55.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:46:55.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:55.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:56.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:57.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:57.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:57.191 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:46:57.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:46:57.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:57.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:57.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:46:58.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:46:58.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:46:58.627 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:46:58.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:46:58.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:46:58.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:46:59.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:00.053 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:47:00.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:47:00.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:00.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:00.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:01.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:01.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:01.553 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:47:01.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:47:01.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:01.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:01.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:02.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:02.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:02.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:47:02.966 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:47:02.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:02.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:03.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:04.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:04.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:04.403 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:47:04.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:47:04.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:04.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:05.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:06.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:06.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:06.052 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:47:06.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:47:06.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:06.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:07.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:08.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:08.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:08.252 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:47:08.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:47:08.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:08.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:08.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:47:09.971 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:47:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:11.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:11.421 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:47:11.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:11.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:47:11.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:11.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:11.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:12.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:12.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:12.942 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:47:12.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:47:12.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:12.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:13.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:14.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:14.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:14.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:47:14.618 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:47:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:14.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:47:16.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:47:16.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:47:16.184 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T03:47:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T03:47:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:47:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:47:16.546 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/49859,v1:127.0.0.1:6819/49859] [v2:127.0.0.1:6820/49859,v1:127.0.0.1:6821/49859] exists,up 9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:47:16.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:47:16.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:47:16.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:47:16.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-09-15T03:47:17.239 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T03:47:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-09-15T03:47:18.036 INFO:tasks.workunit.client.0.smithi138.stderr:noin is set 2024-09-15T03:47:18.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-09-15T03:47:18.933 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T03:47:18.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-09-15T03:47:18.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T03:47:19.437 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T03:47:19.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:47:20.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-09-15T03:47:21.361 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T03:47:21.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-09-15T03:47:21.998 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T03:47:22.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-09-15T03:47:22.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:47:22.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:22.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:47:22.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T03:47:22.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T03:47:22.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T03:47:22.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:22.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:47:22.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672974 2024-09-15T03:47:22.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672974 2024-09-15T03:47:22.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672974' 2024-09-15T03:47:22.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:22.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:47:22.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-15T03:47:22.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-15T03:47:22.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672974 2-64424509446' 2024-09-15T03:47:22.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:22.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T03:47:22.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:22.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:47:22.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T03:47:22.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:22.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T03:47:22.804 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836500 2024-09-15T03:47:22.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T03:47:22.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:47:23.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-15T03:47:23.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:47:24.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:47:24.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:47:24.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T03:47:24.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:24.532 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672974 2024-09-15T03:47:24.532 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:24.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:47:24.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672974 2024-09-15T03:47:24.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:24.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672974 2024-09-15T03:47:24.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T03:47:24.535 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672974 2024-09-15T03:47:24.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:47:24.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672974 -lt 42949672974 2024-09-15T03:47:24.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:24.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-15T03:47:24.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:24.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:47:24.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-15T03:47:24.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:24.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-15T03:47:24.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T03:47:24.929 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509446 2024-09-15T03:47:24.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:47:25.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:47:25.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:47:25.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:47:25.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:47:25.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:47:25.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:47:25.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:47:25.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:47:25.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:47:25.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:47:25.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:47:25.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:47:25.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:47:25.754 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:47:25.754 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:47:25.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:47:25.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:25.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:47:25.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-15T03:47:25.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-15T03:47:25.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T03:47:25.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:25.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:47:25.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-15T03:47:25.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-15T03:47:25.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672977' 2024-09-15T03:47:25.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:25.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672977 2-64424509449' 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-15T03:47:26.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:26.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:47:26.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-15T03:47:26.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:26.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-15T03:47:26.093 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836503 2024-09-15T03:47:26.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T03:47:26.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:47:26.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836501 -lt 21474836503 2024-09-15T03:47:26.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:47:27.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:47:27.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:47:27.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T03:47:27.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:27.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-15T03:47:27.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:27.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:47:27.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-15T03:47:27.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:27.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-15T03:47:27.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T03:47:27.823 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672977 2024-09-15T03:47:27.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:47:28.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T03:47:28.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:28.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-15T03:47:28.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:28.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:47:28.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-15T03:47:28.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:28.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-15T03:47:28.202 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509449 2024-09-15T03:47:28.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-15T03:47:28.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:47:28.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-15T03:47:28.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:47:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:47:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:47:29.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:47:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:47:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:47:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:47:29.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:47:29.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:47:29.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:47:29.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:47:29.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:47:29.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:47:29.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-09-15T03:47:29.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-09-15T03:47:30.278 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:47:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-09-15T03:47:30.289 INFO:tasks.workunit.client.0.smithi138.stdout:primary and soon to be divergent is 1 2024-09-15T03:47:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-09-15T03:47:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:47:30.637 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:47:30.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T03:47:20.518108+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:34:10.960283+0000 0 0 2024-09-15T03:47:30.638 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:47:30.638 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:47:30.638 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-09-15T03:47:30.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-09-15T03:47:30.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-09-15T03:47:30.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-09-15T03:47:30.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-09-15T03:47:30.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-09-15T03:47:30.650 INFO:tasks.workunit.client.0.smithi138.stdout:writing initial objects 2024-09-15T03:47:30.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-09-15T03:47:30.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:30.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-09-15T03:47:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-09-15T03:47:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-09-15T03:47:31.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-09-15T03:47:31.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-09-15T03:47:31.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-09-15T03:47:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-09-15T03:47:31.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-09-15T03:47:31.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-09-15T03:47:31.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:31.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-09-15T03:47:32.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-09-15T03:47:32.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-09-15T03:47:32.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-09-15T03:47:32.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-09-15T03:47:32.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-09-15T03:47:32.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:32.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-09-15T03:47:33.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-09-15T03:47:33.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-09-15T03:47:33.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-09-15T03:47:33.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-09-15T03:47:33.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-09-15T03:47:33.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-09-15T03:47:33.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-09-15T03:47:33.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:33.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-09-15T03:47:34.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-09-15T03:47:34.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-09-15T03:47:34.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-09-15T03:47:34.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-09-15T03:47:34.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-09-15T03:47:34.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-09-15T03:47:34.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:34.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-09-15T03:47:35.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-09-15T03:47:35.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-09-15T03:47:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-09-15T03:47:35.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-09-15T03:47:35.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-09-15T03:47:35.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-09-15T03:47:35.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:35.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-09-15T03:47:36.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-09-15T03:47:36.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-09-15T03:47:36.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-09-15T03:47:36.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-09-15T03:47:36.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-09-15T03:47:36.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:36.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-09-15T03:47:37.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-09-15T03:47:37.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-09-15T03:47:37.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-09-15T03:47:37.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-09-15T03:47:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-09-15T03:47:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-09-15T03:47:38.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-09-15T03:47:38.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-09-15T03:47:38.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-09-15T03:47:38.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-09-15T03:47:38.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-09-15T03:47:38.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-09-15T03:47:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-09-15T03:47:38.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:38.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-09-15T03:47:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-09-15T03:47:39.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-09-15T03:47:39.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-09-15T03:47:39.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-09-15T03:47:39.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-09-15T03:47:39.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-09-15T03:47:39.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:39.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-09-15T03:47:40.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-09-15T03:47:40.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-09-15T03:47:40.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-09-15T03:47:40.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-09-15T03:47:40.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-09-15T03:47:40.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:40.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-09-15T03:47:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-09-15T03:47:41.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-09-15T03:47:41.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-09-15T03:47:41.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-09-15T03:47:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-09-15T03:47:41.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-09-15T03:47:41.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-09-15T03:47:41.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:41.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-09-15T03:47:42.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-09-15T03:47:42.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-09-15T03:47:42.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-09-15T03:47:42.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-09-15T03:47:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-09-15T03:47:42.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:42.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-09-15T03:47:43.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-09-15T03:47:43.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-09-15T03:47:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-09-15T03:47:43.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-09-15T03:47:43.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-09-15T03:47:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-09-15T03:47:43.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-09-15T03:47:43.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:43.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-09-15T03:47:44.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-09-15T03:47:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-09-15T03:47:44.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-09-15T03:47:44.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-09-15T03:47:44.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-09-15T03:47:44.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:44.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-09-15T03:47:45.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-09-15T03:47:45.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:47:45.139 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:47:45.140 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:47:45.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:47:45.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:47:45.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:47:45.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:47:45.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:47:45.546 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:47:45.546 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:47:45.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:47:45.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:45.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:47:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T03:47:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T03:47:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T03:47:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:45.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:47:45.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672983 2024-09-15T03:47:45.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672983 2024-09-15T03:47:45.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672983' 2024-09-15T03:47:45.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:47:45.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:47:45.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-15T03:47:45.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-15T03:47:45.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672983 2-64424509455' 2024-09-15T03:47:45.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:45.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T03:47:45.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:45.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:47:45.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T03:47:45.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:45.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T03:47:45.911 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836509 2024-09-15T03:47:45.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T03:47:45.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:47:46.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T03:47:46.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:46.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672983 2024-09-15T03:47:46.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:46.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:47:46.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672983 2024-09-15T03:47:46.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:46.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672983 2024-09-15T03:47:46.268 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672983 2024-09-15T03:47:46.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-15T03:47:46.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:47:46.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672983 2024-09-15T03:47:46.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:47:47.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:47:47.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:47:48.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-15T03:47:48.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:47:48.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-15T03:47:48.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:47:48.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:47:48.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-15T03:47:48.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:47:48.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-15T03:47:48.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T03:47:48.006 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509455 2024-09-15T03:47:48.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:47:48.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T03:47:48.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:47:48.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:47:48.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:47:48.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:47:48.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:47:49.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:47:49.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:47:49.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:47:49.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:47:49.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-15T03:47:49.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T03:47:50.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-09-15T03:47:50.155 INFO:tasks.workunit.client.0.smithi138.stdout:blackholing osds 0 2 2024-09-15T03:47:50.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-09-15T03:47:50.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:47:50.492 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:47:50.493 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-15T03:47:20.518108+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:34:10.960283+0000 0 0 2024-09-15T03:47:50.493 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:47:50.493 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:47:50.493 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:47:50.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:47:50.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:47:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T03:47:50.575 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:47:50.575 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:47:50.575 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:47:50.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:47:50.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-09-15T03:47:50.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:47:50.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:47:50.584 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:47:50.585 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:47:50.585 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:47:50.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:47:50.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:47:50.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T03:47:50.656 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:47:50.656 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:47:50.656 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:47:50.662 INFO:tasks.workunit.client.0.smithi138.stdout:create new divergent objects 2024-09-15T03:47:50.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-09-15T03:47:50.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-09-15T03:47:50.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:47:50.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:47:50.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-09-15T03:47:50.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-09-15T03:47:50.664 INFO:tasks.workunit.client.0.smithi138.stdout:writing divergent objects 2024-09-15T03:47:50.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-09-15T03:47:50.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-09-15T03:47:50.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T03:47:50.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T03:47:50.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-09-15T03:47:50.666 INFO:tasks.workunit.client.0.smithi138.stdout:remove divergent objects 2024-09-15T03:47:50.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-09-15T03:47:50.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-09-15T03:47:50.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-09-15T03:47:50.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T03:47:50.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-09-15T03:47:50.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-09-15T03:47:50.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T03:47:50.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-09-15T03:47:50.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-09-15T03:47:50.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-09-15T03:47:50.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-09-15T03:47:50.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-09-15T03:48:00.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-09-15T03:48:00.675 INFO:tasks.workunit.client.0.smithi138.stderr:rados: no process found 2024-09-15T03:48:00.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-09-15T03:48:00.676 INFO:tasks.workunit.client.0.smithi138.stdout:killing all the osds 2024-09-15T03:48:00.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:48:01.019 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:48:01.019 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 active+clean 2024-09-15T03:47:20.518108+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:34:10.960283+0000 0 0 2024-09-15T03:48:01.019 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:48:01.020 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:48:01.020 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:48:01.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-09-15T03:48:01.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:48:01.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:48:01.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:48:01.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:48:01.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:48:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:48:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T03:48:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-09-15T03:48:01.756 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T03:48:01.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T03:48:01.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-09-15T03:48:02.305 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T03:48:02.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-09-15T03:48:02.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-09-15T03:48:02.861 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T03:48:02.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:02.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-09-15T03:48:03.389 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T03:48:03.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:03.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-09-15T03:48:03.915 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T03:48:03.925 INFO:tasks.workunit.client.0.smithi138.stdout:bringing up non_divergent 0 2 2024-09-15T03:48:03.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-09-15T03:48:03.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:48:04.260 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:48:04.260 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 46676 0 0 16 90 16 stale+active+clean 2024-09-15T03:47:20.518108+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:34:10.960283+0000 0 0 2024-09-15T03:48:04.260 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:48:04.260 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:48:04.260 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:48:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:48:04.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:48:04.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:48:04.276 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:48:04.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:48:04.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:48:04.303 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:04.302+0000 7f395c82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:04.320 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:04.318+0000 7f395c82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:04.320 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:04.319+0000 7f395c82d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:48:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:48:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:48:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:48:04.840 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:04.839+0000 7f395c82d440 -1 Falling back to public interface 2024-09-15T03:48:05.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:06.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:06.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:06.029 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:48:06.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:48:06.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:06.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:48:06.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:07.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:07.394 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:48:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:48:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:48:07.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:07.727+0000 7f395c82d440 -1 osd.0 29 log_to_monitors true 2024-09-15T03:48:07.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:08.763 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:48:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:48:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:08.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:48:09.125 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/54224,v1:127.0.0.1:6803/54224] [v2:127.0.0.1:6804/54224,v1:127.0.0.1:6805/54224] exists,up 0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:48:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:48:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:48:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:48:09.132 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:48:09.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:48:09.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:48:09.159 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:09.158+0000 7f5c70a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:09.166 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:09.165+0000 7f5c70a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:09.167 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:09.166+0000 7f5c70a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:48:09.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:48:09.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:09.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:48:09.685 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:09.685+0000 7f5c70a0b440 -1 Falling back to public interface 2024-09-15T03:48:09.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:10.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:10.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:10.867 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:48:10.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:48:10.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:10.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:48:11.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:12.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:12.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:12.234 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:48:12.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:48:12.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:12.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:48:12.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:48:12.708 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:48:12.706+0000 7f5c70a0b440 -1 osd.2 29 log_to_monitors true 2024-09-15T03:48:13.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:48:13.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:48:13.622 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:48:13.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:48:13.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:48:13.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:48:13.993 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/54802,v1:127.0.0.1:6811/54802] [v2:127.0.0.1:6812/54802,v1:127.0.0.1:6813/54802] exists,up 9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:48:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:48:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:48:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:48:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-09-15T03:48:14.568 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T03:48:14.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:48:14.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-09-15T03:48:15.088 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already in. 2024-09-15T03:48:15.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:48:15.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-09-15T03:48:15.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:48:15.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:48:15.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:48:15.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:48:15.100 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:48:15.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:48:15.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:48:15.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:48:15.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:48:15.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:48:15.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:48:15.645 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:48:15.645 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:48:15.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:48:15.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:15.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:48:15.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822660 2024-09-15T03:48:15.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822660 2024-09-15T03:48:15.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660' 2024-09-15T03:48:15.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:15.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:48:15.840 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:48:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:48:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:48:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:48:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:15.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:48:15.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757251 2024-09-15T03:48:15.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757251 2024-09-15T03:48:15.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-09-15T03:48:15.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:48:15.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822660 2024-09-15T03:48:15.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:48:15.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:48:15.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822660 2024-09-15T03:48:15.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:48:15.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822660 2024-09-15T03:48:15.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-09-15T03:48:15.957 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 154618822660 2024-09-15T03:48:15.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:48:16.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-15T03:48:16.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:48:17.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:48:17.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:48:17.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822658 -lt 154618822660 2024-09-15T03:48:17.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:48:18.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T03:48:18.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:48:19.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822660 -lt 154618822660 2024-09-15T03:48:19.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:48:19.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757251 2024-09-15T03:48:19.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:48:19.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:48:19.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757251 2024-09-15T03:48:19.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:48:19.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757251 2024-09-15T03:48:19.040 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 163208757251 2024-09-15T03:48:19.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-09-15T03:48:19.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:48:19.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757251 -lt 163208757251 2024-09-15T03:48:19.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:48:19.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:19.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:19.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:48:19.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:19.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:20.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:20.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:20.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:20.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:20.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:48:20.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:20.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:20.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:21.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:21.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:21.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:21.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:21.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:21.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:21.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:21.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:21.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:21.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:21.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:21.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:22.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:22.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:22.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:48:22.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:22.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:22.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:22.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:22.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:22.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:22.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:23.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:23.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:23.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:23.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:23.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:23.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:48:23.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:23.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:48:24.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:24.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:24.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:24.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:24.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:24.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:24.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:24.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:24.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:24.532 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:24.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:24.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:25.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:25.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:25.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:25.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:25.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:48:25.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:25.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:48:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:26.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:26.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:26.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:26.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:26.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:26.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:26.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:26.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:26.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:26.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:27.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:27.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:27.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:27.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:48:27.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:27.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:48:29.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:29.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:29.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:29.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:29.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:29.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:29.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:29.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:29.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:29.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:29.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:30.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:30.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:30.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:48:30.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:30.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:48:33.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:33.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:33.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:34.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:34.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:34.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:34.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:34.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:34.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:35.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:35.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:35.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:48:35.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:35.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:48:41.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:41.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:41.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:41.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:41.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:41.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:41.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:41.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:41.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:41.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:41.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:41.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:42.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:42.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:42.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:42.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:42.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:42.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:48:42.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:42.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:48:50.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:50.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:50.147 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:50.147 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:50.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:50.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:50.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:50.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:50.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:50.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:50.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:50.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:50.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:50.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:50.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:48:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:48:51.933 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:48:51.933 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "c2e16514-f3e8-422a-a2fd-902611d3744d", 2024-09-15T03:48:51.933 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:48:51.932238+0000", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:48:51.934 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:48:51.935 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:48:51.936 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:48:51.937 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:45:30.595927Z", 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:45:30.595927Z", 2024-09-15T03:48:51.938 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:48:51.939 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.940 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.941 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 41, 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:45:31.440499+0000", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:48:15.207556+0000", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:48:12.898876+0000", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:48:14.889947+0000", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:48:51.942 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:48:51.943 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:47:19.202041+0000", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T03:48:51.944 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T03:48:51.945 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "29", 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:48:51.946 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:48:51.947 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:48:51.947 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:48:51.947 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:48:51.947 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T03:48:51.947 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:48:51.948 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:48:51.949 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "0433c468-fffb-4374-a487-5905fddfa18d", 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:48:51.950 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 36, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 40, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 30, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.951 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.952 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.953 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.954 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/54224", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/54224", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/54224", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/54224", 2024-09-15T03:48:51.955 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "c06d852a-9fbb-4f83-aa04-2a1803ed93d0", 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:48:51.956 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 23, 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 31, 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.957 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.958 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.959 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.960 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/48081", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/48081", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48081", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48081", 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.961 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "9081311c-9dfa-4576-811b-7ba7a9f09c12", 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 38, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 0, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 32, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:48:51.962 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.963 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.964 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:48:51.965 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/54802", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/54802", 2024-09-15T03:48:51.966 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54802", 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54802", 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:01.501901+0000", 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:48:51.967 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:03.568476+0000", 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:02.113235+0000", 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:39.138096+0000", 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:48:51.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:02.665518+0000", 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:14.719051+0000", 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:48:51.969 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:48:51.970 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.971 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/54224,v1:127.0.0.1:6805/54224]", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:48:51.972 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:45:37.235395Z", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:48:51.973 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/54224,v1:127.0.0.1:6803/54224]", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/54224,v1:127.0.0.1:6809/54224]", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/54224,v1:127.0.0.1:6807/54224]", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:48:51.974 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/48081,v1:127.0.0.1:6813/48081]", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:48:51.975 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:48:51.976 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:46:11.012125Z", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:48:51.977 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/48081,v1:127.0.0.1:6811/48081]", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48081,v1:127.0.0.1:6817/48081]", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48081,v1:127.0.0.1:6815/48081]", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:48:51.978 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/54802,v1:127.0.0.1:6813/54802]", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:48:51.979 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.980 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:46:46.965497Z", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/54802,v1:127.0.0.1:6811/54802]", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54802,v1:127.0.0.1:6817/54802]", 2024-09-15T03:48:51.981 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54802,v1:127.0.0.1:6815/54802]", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:48:51.982 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [] 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 36 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:48:51.983 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 38 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 41, 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.984 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:48:51.985 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:48:51.986 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:48:51.988 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.989 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:48:51.990 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.991 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.992 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:48:51.993 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:48:51.994 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:48:51.995 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:48:51.996 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:48:51.997 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:48:51.998 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:51.999 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.000 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 1, 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48300, 2024-09-15T03:48:52.001 INFO:tasks.workunit.client.0.smithi138.stderr:report 1990068910 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:48:52.002 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 100, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 100, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:48:52.003 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:48:52.004 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 16, 2024-09-15T03:48:52.005 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 16, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:48:52.006 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894772, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2104, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892597, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678028, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206300672, 2024-09-15T03:48:52.007 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2154496, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 416752, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914019802 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:48:52.008 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:48:52.009 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894772, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2104, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892597, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678028, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:48:52.010 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206300672, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2154496, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 416752, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914019802 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:48:52.011 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.012 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48300, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:48:52.013 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 100, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 100, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:48:52.014 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:48:52.015 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1220608, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 143276, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:48:52.016 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 16, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 16, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 154618822667 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672986 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 163208757258 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:48:52.018 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:48:52.019 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:48:52.020 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 5, 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:48:05.686843+0000", 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:48:52.021 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 524, 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:48:52.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:48:52.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-09-15T03:48:52.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-09-15T03:48:52.023 INFO:tasks.workunit.client.0.smithi138.stdout:writing non-divergent object existing_4 2024-09-15T03:48:52.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-09-15T03:48:52.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:48:52.325 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:48:52.325 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 29'100 41:124 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:48:52.325 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:48:52.325 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:48:52.325 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:48:52.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:48:52.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:48:52.456 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:48:52.456 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:48:52.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:48:52.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:48:52.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:48:52.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:48:52.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:48:52.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:48:52.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:48:52.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:48:52.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:52.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:48:52.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822669 2024-09-15T03:48:52.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822669 2024-09-15T03:48:52.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822669' 2024-09-15T03:48:52.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:52.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:48:53.059 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:48:53.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:48:53.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:48:53.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:48:53.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:48:53.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757260 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757260 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822669 2-163208757260' 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822669 2024-09-15T03:48:53.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:48:53.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:48:53.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822669 2024-09-15T03:48:53.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:48:53.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822669 2024-09-15T03:48:53.175 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 154618822669 2024-09-15T03:48:53.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822669' 2024-09-15T03:48:53.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:48:53.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822667 -lt 154618822669 2024-09-15T03:48:53.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:48:54.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:48:54.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:48:54.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822669 -lt 154618822669 2024-09-15T03:48:54.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:48:54.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757260 2024-09-15T03:48:54.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:48:54.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:48:54.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757260 2024-09-15T03:48:54.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:48:54.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757260 2024-09-15T03:48:54.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757260' 2024-09-15T03:48:54.946 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 163208757260 2024-09-15T03:48:54.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:48:55.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757260 -lt 163208757260 2024-09-15T03:48:55.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:48:55.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:55.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:55.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:48:55.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:55.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:56.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:56.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:56.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:56.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:56.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:56.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:48:56.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:48:56.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:48:56.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:56.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:48:56.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:56.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:56.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:56.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:56.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:56.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:56.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:56.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:57.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:57.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:57.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:57.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:57.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:57.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:58.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:58.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:58.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:48:58.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:58.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:58.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:48:58.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:48:58.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:48:58.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:48:58.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:48:59.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:48:59.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:48:59.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:48:59.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:48:59.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:48:59.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:48:59.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:48:59.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:00.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:00.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:00.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:00.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:00.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:00.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:00.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:00.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:00.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:00.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:00.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:00.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:00.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:49:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:49:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:49:02.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:49:02.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:49:02.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:02.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:02.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:02.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:02.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:02.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:03.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:03.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:03.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:49:03.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:49:03.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:49:04.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:49:04.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:49:04.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:49:04.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:49:04.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:49:04.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:49:04.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:49:04.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:05.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:05.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:05.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:05.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:05.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:06.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:06.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:06.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:49:06.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:49:06.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:49:09.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:49:09.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:49:09.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:49:09.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:49:09.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:49:09.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:49:09.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:49:09.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:09.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:09.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:09.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:09.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:49:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:49:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:49:17.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:49:17.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:49:17.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:17.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:17.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:18.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:18.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:18.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:49:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:49:18.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:49:25.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:49:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:49:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:49:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:49:25.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:49:25.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:49:25.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:49:25.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:49:26.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:49:26.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:49:26.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:49:26.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:49:26.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:49:27.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:49:27.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:49:27.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:49:27.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "c2e16514-f3e8-422a-a2fd-902611d3744d", 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:49:27.781151+0000", 2024-09-15T03:49:27.782 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:49:27.783 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:49:27.784 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 2 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is stuck undersized for 70s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:49:27.785 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.786 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:45:30.595927Z", 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:45:30.595927Z", 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:49:27.787 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:49:27.788 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.789 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:49:27.790 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:49:27.790 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.790 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.790 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.790 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 41, 2024-09-15T03:49:27.791 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:45:31.440499+0000", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:48:15.207556+0000", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:48:12.898876+0000", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:48:14.889947+0000", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:49:27.792 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:49:27.793 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:47:19.202041+0000", 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:49:27.794 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:49:27.795 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "29", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:49:27.796 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:49:27.797 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:49:27.798 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "0433c468-fffb-4374-a487-5905fddfa18d", 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 36, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 40, 2024-09-15T03:49:27.799 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 30, 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.800 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.801 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.802 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.803 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54224 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/54224", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/54224", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/54224", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/54224", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.804 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "c06d852a-9fbb-4f83-aa04-2a1803ed93d0", 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 23, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 31, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:49:27.805 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.807 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.808 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 48081 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/48081", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/48081", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48081", 2024-09-15T03:49:27.809 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48081", 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "9081311c-9dfa-4576-811b-7ba7a9f09c12", 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:49:27.810 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 29, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 38, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 0, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 32, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.811 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.812 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.813 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.814 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 54802 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/54802", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/54802", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/54802", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/54802", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:49:27.815 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:01.501901+0000", 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:03.568476+0000", 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:49:27.816 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:02.113235+0000", 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:46:39.138096+0000", 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.817 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:48:02.665518+0000", 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:47:14.719051+0000", 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.818 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:49:27.819 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.820 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/54224,v1:127.0.0.1:6805/54224]", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:49:27.821 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:49:27.822 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:45:37.235395Z", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/54224,v1:127.0.0.1:6803/54224]", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:49:27.823 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/54224,v1:127.0.0.1:6809/54224]", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/54224,v1:127.0.0.1:6807/54224]", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:49:27.824 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/48081,v1:127.0.0.1:6813/48081]", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:49:27.825 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:49:27.826 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:46:11.012125Z", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/48081,v1:127.0.0.1:6811/48081]", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48081,v1:127.0.0.1:6817/48081]", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48081,v1:127.0.0.1:6815/48081]", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:49:27.827 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:49:27.828 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/54802,v1:127.0.0.1:6813/54802]", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:49:27.829 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:46:46.965497Z", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:49:27.830 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/54802,v1:127.0.0.1:6811/54802]", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/54802,v1:127.0.0.1:6817/54802]", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/54802,v1:127.0.0.1:6815/54802]", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:49:27.831 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [] 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:49:27.832 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 36 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 38 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 41, 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:49:27.833 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:49:27.834 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.835 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.836 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:49:27.837 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:49:27.838 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:49:27.839 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.840 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.841 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:49:27.842 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.843 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:49:27.844 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:49:27.845 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:49:27.846 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:49:27.847 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:49:27.848 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:49:27.849 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 1, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:49:27.850 INFO:tasks.workunit.client.0.smithi138.stderr:report 3376364721 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 47971, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:49:27.851 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:49:27.852 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:49:27.853 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:49:27.854 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 17, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 17, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:49:27.855 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894772, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2104, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892597, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678028, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206300672, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2154496, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 416094, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:49:27.856 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914019802 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:49:27.857 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:49:27.858 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894772, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2104, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892597, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678028, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206300672, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2154496, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 416094, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:49:27.859 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914019802 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:49:27.860 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:49:27.861 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 47971, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:49:27.862 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:49:27.863 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:49:27.864 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1220608, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 142618, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:49:27.865 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 17, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 17, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 2, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 2, 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:49:27.866 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 154618822676 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:49:27.867 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672986 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 163208757267 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:49:27.868 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:49:27.869 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:49:27.870 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 7, 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:49:15.699576+0000", 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 608, 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:49:27.871 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:49:27.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:49:27.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-09-15T03:49:27.872 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery 2024-09-15T03:49:27.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:28.163 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:28.163 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:28.163 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:28.163 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:28.163 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:28.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:49:28.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-09-15T03:49:28.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:49:28.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:49:28.175 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:49:28.175 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:49:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:49:28.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:49:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:49:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T03:49:28.249 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:49:28.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:49:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T03:49:28.333 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:49:28.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-09-15T03:49:28.339 INFO:tasks.workunit.client.0.smithi138.stdout:revive divergent 1 2024-09-15T03:49:28.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:28.674 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:28.674 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:28.674 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:28.674 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:28.674 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:28.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-09-15T03:49:29.336 INFO:tasks.workunit.client.0.smithi138.stderr:noup is set 2024-09-15T03:49:29.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-15T03:49:29.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:49:29.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:49:29.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:49:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:49:29.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:49:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:49:29.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:49:29.357 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:49:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:49:29.379 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:29.379+0000 7fbf4560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:29.401 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:29.400+0000 7fbf4560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:29.402 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:29.401+0000 7fbf4560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:29.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-15T03:49:29.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-09-15T03:49:29.915 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:29.915+0000 7fbf4560c440 -1 Falling back to public interface 2024-09-15T03:49:32.929 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:32.928+0000 7fbf4560c440 -1 osd.1 29 log_to_monitors true 2024-09-15T03:49:34.738 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery divergent 2024-09-15T03:49:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-09-15T03:49:34.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:35.081 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:35.081 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 41:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:35.081 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:35.081 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:35.081 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:49:35.092 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:49:35.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:49:35.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:49:35.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T03:49:35.167 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:49:35.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-09-15T03:49:35.997 INFO:tasks.workunit.client.0.smithi138.stderr:noup is unset 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:49:36.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:49:36.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:49:36.010 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:49:36.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:49:36.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:49:36.428 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/54224,v1:127.0.0.1:6803/54224] [v2:127.0.0.1:6804/54224,v1:127.0.0.1:6805/54224] exists,up 0433c468-fffb-4374-a487-5905fddfa18d 2024-09-15T03:49:36.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:49:36.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:49:36.430 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:49:36.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:49:36.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:49:36.894 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57391,v1:127.0.0.1:6819/57391] [v2:127.0.0.1:6820/57391,v1:127.0.0.1:6821/57391] exists,up c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:49:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:49:36.896 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:49:36.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:49:36.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:49:37.263 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/54802,v1:127.0.0.1:6811/54802] [v2:127.0.0.1:6812/54802,v1:127.0.0.1:6813/54802] exists,up 9081311c-9dfa-4576-811b-7ba7a9f09c12 2024-09-15T03:49:37.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:49:37.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:49:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:49:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:37.598 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:37.599 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 43:128 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:37.599 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:37.599 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:37.599 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:37.609 INFO:tasks.workunit.client.0.smithi138.stdout:wait for peering 2024-09-15T03:49:37.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-09-15T03:49:37.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:37.943 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:37.944 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 43:128 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:37.944 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:37.944 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:37.944 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:37.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-09-15T03:49:38.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-09-15T03:49:38.088 INFO:tasks.workunit.client.0.smithi138.stdout:killing divergent 1 2024-09-15T03:49:38.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:38.422 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:38.422 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 47971 0 0 17 84 17 active+undersized+degraded 2024-09-15T03:48:15.457579+0000 41'101 43:128 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:29:17.958306+0000 0 0 2024-09-15T03:49:38.422 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:38.422 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:38.423 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:38.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T03:49:38.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:49:38.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:49:38.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:49:38.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:49:38.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.32635.out 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.32635.out 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:49:38.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T03:49:38.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.32635.out 2024-09-15T03:49:40.052 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.0 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-15T03:49:40.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:023dd3d2:::existing_12:head# 2024-09-15T03:49:40.087 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:04c40a0c:::existing_93:head# 2024-09-15T03:49:40.097 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:064ff5fb:::existing_36:head# 2024-09-15T03:49:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:091f9f1d:::existing_94:head# 2024-09-15T03:49:40.111 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a865a93:::existing_54:head# 2024-09-15T03:49:40.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0fed513c:::existing_41:head# 2024-09-15T03:49:40.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-09-15T03:49:40.152 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:169157fa:::existing_81:head# 2024-09-15T03:49:40.154 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:17731382:::existing_58:head# 2024-09-15T03:49:40.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:18a45a39:::existing_43:head# 2024-09-15T03:49:40.169 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1c160f01:::existing_66:head# 2024-09-15T03:49:40.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1dd6c249:::existing_97:head# 2024-09-15T03:49:40.178 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:227e0c77:::existing_7:head# 2024-09-15T03:49:40.203 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:23e1b063:::existing_59:head# 2024-09-15T03:49:40.210 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:28566edb:::existing_78:head# 2024-09-15T03:49:40.211 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2a101182:::existing_20:head# 2024-09-15T03:49:40.211 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2d1519da:::existing_96:head# 2024-09-15T03:49:40.211 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3078b87a:::existing_86:head# 2024-09-15T03:49:40.212 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:30bde01b:::existing_15:head# 2024-09-15T03:49:40.212 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:317a7e19:::existing_56:head# 2024-09-15T03:49:40.212 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-09-15T03:49:40.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3e16d84b:::existing_48:head# 2024-09-15T03:49:40.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:45fe8203:::existing_75:head# 2024-09-15T03:49:40.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-09-15T03:49:40.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5298dcdb:::existing_57:head# 2024-09-15T03:49:40.214 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52ebaa44:::existing_32:head# 2024-09-15T03:49:40.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:53588912:::existing_9:head# 2024-09-15T03:49:40.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-09-15T03:49:40.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5da0255c:::existing_13:head# 2024-09-15T03:49:40.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5da2dab8:::existing_53:head# 2024-09-15T03:49:40.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5ee3514b:::existing_30:head# 2024-09-15T03:49:40.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:602f83fe:::foo:head# 2024-09-15T03:49:40.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:60426cc7:::existing_62:head# 2024-09-15T03:49:40.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:628918ca:::existing_79:head# 2024-09-15T03:49:40.217 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-09-15T03:49:40.217 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:67e1780f:::existing_27:head# 2024-09-15T03:49:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68143ca1:::existing_71:head# 2024-09-15T03:49:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6bac9509:::existing_29:head# 2024-09-15T03:49:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6bbec798:::existing_45:head# 2024-09-15T03:49:40.219 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6cb23a40:::existing_65:head# 2024-09-15T03:49:40.219 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7055ec5e:::existing_83:head# 2024-09-15T03:49:40.219 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:70a54e2b:::existing_87:head# 2024-09-15T03:49:40.219 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:750ace21:::existing_19:head# 2024-09-15T03:49:40.220 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:78623b1d:::existing_6:head# 2024-09-15T03:49:40.229 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:79881152:::existing_63:head# 2024-09-15T03:49:40.230 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-09-15T03:49:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81490677:::existing_76:head# 2024-09-15T03:49:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-09-15T03:49:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-09-15T03:49:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-09-15T03:49:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:923c5d4d:::existing_5:head# 2024-09-15T03:49:40.236 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:92731e71:::existing_89:head# 2024-09-15T03:49:40.238 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:928205cf:::existing_82:head# 2024-09-15T03:49:40.239 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:93be2267:::existing_24:head# 2024-09-15T03:49:40.239 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:99ceda2c:::existing_44:head# 2024-09-15T03:49:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9dd917ee:::existing_26:head# 2024-09-15T03:49:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a28805a6:::existing_72:head# 2024-09-15T03:49:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a294a8f6:::existing_16:head# 2024-09-15T03:49:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a2f5473a:::existing_42:head# 2024-09-15T03:49:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a504a533:::existing_90:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a691337e:::existing_61:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a9fde86e:::existing_18:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aa309faa:::existing_68:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b279b1fe:::existing_10:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b343e277:::existing_88:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b7130e90:::existing_80:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bc6f7309:::existing_70:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:be8489b7:::existing_38:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bebec877:::existing_74:head# 2024-09-15T03:49:40.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c04347d8:::existing_60:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c156033e:::existing_100:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ccb3865b:::existing_73:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd84e19e:::existing_40:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d41a25c9:::existing_22:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7f088da:::existing_91:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-15T03:49:40.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e0e22077:::existing_21:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e255b710:::existing_55:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eff1d5db:::existing_14:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f1b156bf:::existing_23:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5ce4199:::existing_34:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f6809456:::existing_11:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f699a2a9:::existing_67:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f9baac43:::existing_25:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fb76688e:::existing_51:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fbb92eec:::existing_69:head# 2024-09-15T03:49:40.250 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T03:49:40.251 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T03:49:40.251 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T03:49:40.251 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T03:49:40.528 INFO:tasks.workunit.client.0.smithi138.stderr:Remove successful 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.32635.out 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T03:49:41.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.32635.out 2024-09-15T03:49:42.048 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={23=1}} deleted_pools ) 2024-09-15T03:49:42.048 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T03:49:42.048 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:023dd3d2:::existing_12:head# 2024-09-15T03:49:42.049 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.190 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04c40a0c:::existing_93:head# 2024-09-15T03:49:42.190 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.249 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:064ff5fb:::existing_36:head# 2024-09-15T03:49:42.249 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.307 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:091f9f1d:::existing_94:head# 2024-09-15T03:49:42.307 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.425 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a865a93:::existing_54:head# 2024-09-15T03:49:42.425 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.517 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0fed513c:::existing_41:head# 2024-09-15T03:49:42.517 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.552 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-09-15T03:49:42.553 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.586 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:169157fa:::existing_81:head# 2024-09-15T03:49:42.586 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.619 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:17731382:::existing_58:head# 2024-09-15T03:49:42.620 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.653 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:18a45a39:::existing_43:head# 2024-09-15T03:49:42.653 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.686 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1c160f01:::existing_66:head# 2024-09-15T03:49:42.687 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.720 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dd6c249:::existing_97:head# 2024-09-15T03:49:42.720 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.753 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:227e0c77:::existing_7:head# 2024-09-15T03:49:42.753 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:23e1b063:::existing_59:head# 2024-09-15T03:49:42.787 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.820 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:28566edb:::existing_78:head# 2024-09-15T03:49:42.820 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.854 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2a101182:::existing_20:head# 2024-09-15T03:49:42.854 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.887 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2d1519da:::existing_96:head# 2024-09-15T03:49:42.887 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.921 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3078b87a:::existing_86:head# 2024-09-15T03:49:42.921 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.954 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30bde01b:::existing_15:head# 2024-09-15T03:49:42.954 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:42.987 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:317a7e19:::existing_56:head# 2024-09-15T03:49:42.988 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.021 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-09-15T03:49:43.021 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.054 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3e16d84b:::existing_48:head# 2024-09-15T03:49:43.055 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.088 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45fe8203:::existing_75:head# 2024-09-15T03:49:43.088 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.121 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-09-15T03:49:43.122 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.155 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5298dcdb:::existing_57:head# 2024-09-15T03:49:43.155 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.188 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52ebaa44:::existing_32:head# 2024-09-15T03:49:43.189 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.222 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:53588912:::existing_9:head# 2024-09-15T03:49:43.222 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.255 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-09-15T03:49:43.255 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.289 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5da0255c:::existing_13:head# 2024-09-15T03:49:43.289 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.322 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5da2dab8:::existing_53:head# 2024-09-15T03:49:43.322 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.356 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5ee3514b:::existing_30:head# 2024-09-15T03:49:43.356 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.389 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:602f83fe:::foo:head# 2024-09-15T03:49:43.389 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:60426cc7:::existing_62:head# 2024-09-15T03:49:43.423 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.456 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:628918ca:::existing_79:head# 2024-09-15T03:49:43.456 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.489 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-09-15T03:49:43.490 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.523 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:67e1780f:::existing_27:head# 2024-09-15T03:49:43.523 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.556 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68143ca1:::existing_71:head# 2024-09-15T03:49:43.556 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.590 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bac9509:::existing_29:head# 2024-09-15T03:49:43.590 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.648 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bbec798:::existing_45:head# 2024-09-15T03:49:43.649 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.682 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6cb23a40:::existing_65:head# 2024-09-15T03:49:43.682 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.715 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7055ec5e:::existing_83:head# 2024-09-15T03:49:43.715 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.749 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:70a54e2b:::existing_87:head# 2024-09-15T03:49:43.749 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.791 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:750ace21:::existing_19:head# 2024-09-15T03:49:43.791 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:78623b1d:::existing_6:head# 2024-09-15T03:49:43.849 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.883 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:79881152:::existing_63:head# 2024-09-15T03:49:43.883 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.916 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-09-15T03:49:43.917 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.949 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81490677:::existing_76:head# 2024-09-15T03:49:43.950 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:43.983 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-09-15T03:49:43.984 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.017 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-09-15T03:49:44.017 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.050 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-09-15T03:49:44.051 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.092 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:923c5d4d:::existing_5:head# 2024-09-15T03:49:44.092 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.134 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92731e71:::existing_89:head# 2024-09-15T03:49:44.134 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.168 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:928205cf:::existing_82:head# 2024-09-15T03:49:44.168 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.201 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:93be2267:::existing_24:head# 2024-09-15T03:49:44.201 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.234 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:99ceda2c:::existing_44:head# 2024-09-15T03:49:44.234 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.268 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9dd917ee:::existing_26:head# 2024-09-15T03:49:44.268 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.301 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a28805a6:::existing_72:head# 2024-09-15T03:49:44.301 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.368 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a294a8f6:::existing_16:head# 2024-09-15T03:49:44.368 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.419 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a2f5473a:::existing_42:head# 2024-09-15T03:49:44.419 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.486 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a504a533:::existing_90:head# 2024-09-15T03:49:44.486 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.519 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a691337e:::existing_61:head# 2024-09-15T03:49:44.519 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.553 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-09-15T03:49:44.553 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.586 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a9fde86e:::existing_18:head# 2024-09-15T03:49:44.586 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.620 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aa309faa:::existing_68:head# 2024-09-15T03:49:44.620 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.653 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-09-15T03:49:44.653 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.720 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b279b1fe:::existing_10:head# 2024-09-15T03:49:44.720 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.753 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b343e277:::existing_88:head# 2024-09-15T03:49:44.754 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7130e90:::existing_80:head# 2024-09-15T03:49:44.787 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.820 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bc6f7309:::existing_70:head# 2024-09-15T03:49:44.820 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.854 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:be8489b7:::existing_38:head# 2024-09-15T03:49:44.854 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.887 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-09-15T03:49:44.888 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.921 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bebec877:::existing_74:head# 2024-09-15T03:49:44.921 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.954 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c04347d8:::existing_60:head# 2024-09-15T03:49:44.954 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:44.988 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c156033e:::existing_100:head# 2024-09-15T03:49:44.988 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.021 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-09-15T03:49:45.021 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-09-15T03:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.105 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-09-15T03:49:45.105 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.138 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-09-15T03:49:45.138 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.172 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ccb3865b:::existing_73:head# 2024-09-15T03:49:45.172 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.205 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd84e19e:::existing_40:head# 2024-09-15T03:49:45.205 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.239 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-09-15T03:49:45.239 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.272 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d41a25c9:::existing_22:head# 2024-09-15T03:49:45.272 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.305 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7f088da:::existing_91:head# 2024-09-15T03:49:45.306 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.339 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-15T03:49:45.339 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.373 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-15T03:49:45.373 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.407 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-15T03:49:45.407 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.440 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-15T03:49:45.440 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.474 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e0e22077:::existing_21:head# 2024-09-15T03:49:45.474 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e255b710:::existing_55:head# 2024-09-15T03:49:45.507 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.540 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eff1d5db:::existing_14:head# 2024-09-15T03:49:45.541 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.574 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f1b156bf:::existing_23:head# 2024-09-15T03:49:45.574 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.608 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5ce4199:::existing_34:head# 2024-09-15T03:49:45.608 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.641 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f6809456:::existing_11:head# 2024-09-15T03:49:45.641 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.674 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f699a2a9:::existing_67:head# 2024-09-15T03:49:45.674 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.708 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f9baac43:::existing_25:head# 2024-09-15T03:49:45.708 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.741 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fb76688e:::existing_51:head# 2024-09-15T03:49:45.741 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.781 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fbb92eec:::existing_69:head# 2024-09-15T03:49:45.781 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:49:45.841 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 47 info 1.0( v 47'102 lc 0'0 (29'90,47'102] local-lis/les=46/47 n=101 ec=23/23 lis/c=46/23 les/c/f=47/24/0 sis=46) 2024-09-15T03:49:45.892 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T03:49:46.412 INFO:tasks.workunit.client.0.smithi138.stdout:reviving divergent 1 2024-09-15T03:49:46.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-09-15T03:49:46.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:49:46.756 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:49:46.756 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-15T03:49:36.628784+0000 47'102 47:129 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:34:33.862432+0000 0 0 2024-09-15T03:49:46.756 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:49:46.756 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:49:46.756 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:49:46.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:49:46.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:49:46.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:49:46.772 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:49:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:49:46.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:49:46.799 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:46.798+0000 7f9aebc2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:46.801 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:46.800+0000 7f9aebc2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:46.802 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:46.801+0000 7f9aebc2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:49:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:49:47.127 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:49:47.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:49:47.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:49:47.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:47.319+0000 7f9aebc2f440 -1 Falling back to public interface 2024-09-15T03:49:47.513 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57391,v1:127.0.0.1:6819/57391] [v2:127.0.0.1:6820/57391,v1:127.0.0.1:6821/57391] exists,up c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:49:47.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:49:47.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:49:47.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:49:47.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-09-15T03:49:47.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:49:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:49:47.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:49:47.891 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/57391,v1:127.0.0.1:6819/57391] [v2:127.0.0.1:6820/57391,v1:127.0.0.1:6821/57391] exists,up c06d852a-9fbb-4f83-aa04-2a1803ed93d0 2024-09-15T03:49:47.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:49:47.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:49:47.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:49:47.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-09-15T03:49:50.240 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:49:50.239+0000 7f9aebc2f440 -1 osd.1 47 log_to_monitors true 2024-09-15T03:50:07.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-09-15T03:50:07.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-09-15T03:50:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:50:07.976 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:50:07.982 INFO:tasks.workunit.client.0.smithi138.stdout: "ops": [], 2024-09-15T03:50:07.982 INFO:tasks.workunit.client.0.smithi138.stdout: "num_ops": 0 2024-09-15T03:50:07.982 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:50:07.983 INFO:tasks.workunit.client.0.smithi138.stdout:allowing recovery 2024-09-15T03:50:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-09-15T03:50:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:50:08.317 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:50:08.317 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-15T03:49:52.277712+0000 47'102 52:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:43:11.644316+0000 0 0 2024-09-15T03:50:08.317 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:50:08.317 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:50:08.317 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:50:08.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T03:50:08.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T03:50:08.434 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:50:08.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T03:50:08.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T03:50:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:50:08.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-09-15T03:50:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T03:50:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:50:08.669 INFO:tasks.workunit.client.0.smithi138.stdout:reading divergent objects 2024-09-15T03:50:08.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-09-15T03:50:08.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-09-15T03:50:09.003 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:50:09.003 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48454 0 0 12 90 12 active+clean 2024-09-15T03:49:52.277712+0000 47'102 52:152 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:47:19.267442+0000 0'0 2024-09-15T03:47:19.267442+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:43:11.644316+0000 0 0 2024-09-15T03:50:09.003 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:50:09.003 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:50:09.003 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:50:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-09-15T03:50:09.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-09-15T03:50:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:50:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T03:50:09.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:50:09.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T03:50:09.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:50:09.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T03:50:09.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:50:09.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T03:50:09.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-09-15T03:50:09.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:50:09.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-15T03:50:09.166 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-15T03:50:09.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:50:09.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-15T03:50:09.196 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-15T03:50:09.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-09-15T03:50:09.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T03:50:09.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:50:09.204 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4643.0:1 2024-09-15T03:47:50.715458+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:50:09.204 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4643.0:1 2024-09-15T03:47:50.715458+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:50:09.204 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:50:09.204 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2024-09-15T03:50:09.204 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: more recent entry found: 41'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4762.0:1 2024-09-15T03:48:52.399889+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4635.0:1 2024-09-15T03:47:50.708587+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:75ac5091:::newobject_2:head by client.4635.0:1 2024-09-15T03:47:50.708587+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-09-15T03:47:50.714108+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-09-15T03:47:50.714108+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-15T03:50:09.205 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4642.0:1 2024-09-15T03:47:50.714108+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'90 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-15T03:47:50.697663+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-15T03:47:50.697663+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2024-09-15T03:50:09.206 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4631.0:1 2024-09-15T03:47:50.697663+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'90 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-15T03:47:50.710399+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-15T03:47:50.710399+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.207 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'104 (29'1) modify 1:d858c035:::existing_1:head by client.4638.0:1 2024-09-15T03:47:50.710399+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2024-09-15T03:50:09.208 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4630.0:1 2024-09-15T03:47:50.696547+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 20 _merge_object_divergent_entries: keeping 29'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4630.0:1 2024-09-15T03:47:50.696547+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:49:36.215+0000 7fbf285c7640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-15T03:50:09.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T03:50:09.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:50:09.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-09-15T03:50:09.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:50:09.216 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T03:50:09.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-09-15T03:50:09.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.32635.out 2024-09-15T03:50:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.32635.out': No such file or directory 2024-09-15T03:50:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-09-15T03:50:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T03:50:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T03:50:09.747 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:50:09.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:50:10.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:50:10.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:50:10.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:50:10.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:50:10.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:50:10.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:50:10.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:50:10.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:50:10.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:50:10.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:50:10.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:50:10.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:50:10.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:50:10.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:50:10.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:50:10.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:50:10.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:50:10.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:50:10.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:50:10.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:50:10.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:50:10.104 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:50:10.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:50:10.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:50:10.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:50:10.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:50:10.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:50:10.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:50:10.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:50:10.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:50:10.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:50:10.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:50:10.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:50:10.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:50:10.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:50:10.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:50:10.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-15T03:50:10.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:50:10.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32635 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-09-15T03:50:10.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-09-15T03:50:10.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-09-15T03:50:10.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-09-15T03:50:10.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-09-15T03:50:10.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-09-15T03:50:10.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-09-15T03:50:10.116 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-15T03:50:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:50:10.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:50:10.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:50:10.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:50:10.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:50:10.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:50:10.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:50:10.756 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:50:10.756 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:50:10.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get fsid 2024-09-15T03:50:10.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:50:10.832 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:50:10.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get mon_host 2024-09-15T03:50:10.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-09-15T03:50:10.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-15T03:50:10.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:50:10.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:50:10.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:50:10.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-15T03:50:10.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:11.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:50:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:50:11.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T03:50:11.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-09-15T03:50:11.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:50:11.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:50:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:50:11.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:50:11.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:50:11.784 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:50:11.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 516e25a2-58bf-4649-896b-6cdc14377ee5' 2024-09-15T03:50:11.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:50:11.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBzWeZmcJ21LxAAGAXnJ/Qp+wH7piLQEB+cHg== 2024-09-15T03:50:11.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBzWeZmcJ21LxAAGAXnJ/Qp+wH7piLQEB+cHg=="}' 2024-09-15T03:50:11.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 516e25a2-58bf-4649-896b-6cdc14377ee5 -i td/divergent-priors/0/new.json 2024-09-15T03:50:12.094 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:50:12.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T03:50:12.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBzWeZmcJ21LxAAGAXnJ/Qp+wH7piLQEB+cHg== --osd-uuid 516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:50:12.136 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.135+0000 7f400b215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:12.139 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.138+0000 7f400b215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:12.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.139+0000 7f400b215440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:12.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.140+0000 7f400b215440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:50:12.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.140+0000 7f400b215440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:50:12.141 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.140+0000 7f400b215440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:50:12.180 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:12.179+0000 7f400b215440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T03:50:17.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T03:50:17.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:50:17.490 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:50:17.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:50:17.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:50:17.979 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:50:17.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:50:17.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:50:17.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:50:17.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:50:17.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:50:18.008 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:18.007+0000 7fbd83c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:18.009 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:18.009+0000 7fbd83c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:18.011 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:18.010+0000 7fbd83c0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:18.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:50:18.380 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:50:18.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:18.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:18.540 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:18.540+0000 7fbd83c0f440 -1 Falling back to public interface 2024-09-15T03:50:18.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:19.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:19.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:19.738 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:50:19.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:50:19.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:19.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:20.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:21.015 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:21.014+0000 7fbd83c0f440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:50:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:50:21.105 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:50:21.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:21.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:21.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:22.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:22.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:22.628 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:50:22.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:50:22.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:22.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:23.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:24.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:24.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:24.022 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:50:24.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:50:24.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:24.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:24.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:25.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:25.425 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:50:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:50:25.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:25.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:25.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:26.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:26.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:26.841 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:50:26.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:50:26.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:26.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:27.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:28.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:28.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:28.230 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:50:28.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:50:28.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:28.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:28.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:29.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:29.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:29.712 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:50:29.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:50:29.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:29.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:31.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:31.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:31.095 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:50:31.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:50:31.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:31.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:32.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:32.488 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:50:32.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:32.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:50:32.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:32.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:33.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:34.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:34.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:34.283 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:50:34.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:50:34.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:34.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:34.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:35.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:35.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:35.996 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:50:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:50:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:36.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:37.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:37.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:37.891 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:50:37.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:50:37.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:37.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:38.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:39.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:39.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:39.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:50:39.973 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:50:39.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:39.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:40.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:41.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:41.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:50:41.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:50:41.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:41.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:42.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:43.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:43.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:43.418 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:50:43.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:50:43.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:43.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:43.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:44.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:44.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:44.911 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:50:44.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:50:44.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:44.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:50:45.269 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/60809,v1:127.0.0.1:6803/60809] [v2:127.0.0.1:6804/60809,v1:127.0.0.1:6805/60809] exists,up 516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:50:45.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:50:45.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:50:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dd75342f-e3b1-4fab-8134-909af98dedc4' 2024-09-15T03:50:45.276 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:50:45.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:50:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVWeZmce9oERAAlKIL4WJJsxT9F/UF5oDEgQ== 2024-09-15T03:50:45.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVWeZmce9oERAAlKIL4WJJsxT9F/UF5oDEgQ=="}' 2024-09-15T03:50:45.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd75342f-e3b1-4fab-8134-909af98dedc4 -i td/divergent-priors/1/new.json 2024-09-15T03:50:45.795 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:50:45.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T03:50:45.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVWeZmce9oERAAlKIL4WJJsxT9F/UF5oDEgQ== --osd-uuid dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:50:45.838 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.837+0000 7fe96431f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:45.839 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.839+0000 7fe96431f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:45.840 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.840+0000 7fe96431f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.840+0000 7fe96431f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:50:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.840+0000 7fe96431f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:50:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.840+0000 7fe96431f440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:50:45.886 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:45.885+0000 7fe96431f440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T03:50:51.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T03:50:51.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:50:51.448 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T03:50:51.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:50:51.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:50:51.940 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:50:51.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:50:51.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:50:51.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:50:51.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:50:51.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:50:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:51.969+0000 7f6abce4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:51.972 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:51.971+0000 7f6abce4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:51.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:51.972+0000 7f6abce4b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:50:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:50:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:50:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:50:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:52.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:52.499 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:52.498+0000 7f6abce4b440 -1 Falling back to public interface 2024-09-15T03:50:52.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:53.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:53.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:53.653 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:50:53.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:50:53.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:53.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:54.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:54.972 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:50:54.970+0000 7f6abce4b440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:50:55.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:55.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:55.030 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:50:55.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:50:55.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:55.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:55.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:56.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:56.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:56.607 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:50:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:50:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:57.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:58.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:58.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:58.021 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:50:58.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:50:58.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:58.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:58.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:50:59.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:50:59.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:50:59.494 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:50:59.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:50:59.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:50:59.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:50:59.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:00.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:00.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:00.941 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:51:00.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:51:00.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:00.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:01.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:02.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:02.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:02.357 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:51:02.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:51:02.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:02.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:02.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:03.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:03.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:03.813 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:51:03.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:51:03.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:03.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:04.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:05.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:05.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:05.264 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:51:05.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:51:05.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:05.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:05.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:06.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:06.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:06.749 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:51:06.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:51:06.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:06.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:07.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:08.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:08.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:08.118 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:51:08.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:51:08.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:08.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:10.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:10.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:10.056 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:51:10.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:51:10.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:10.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:10.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:11.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:11.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:11.807 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:51:11.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:51:11.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:11.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:13.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:14.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:14.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:14.028 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:51:14.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:51:14.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:14.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:14.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:15.843 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:51:15.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:51:15.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:15.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:16.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:17.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:17.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:17.894 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:51:17.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:51:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:51:18.270 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/62563,v1:127.0.0.1:6811/62563] [v2:127.0.0.1:6812/62563,v1:127.0.0.1:6813/62563] exists,up dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:51:18.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:51:18.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:51:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:51:18.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:51:18.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:51:18.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 3d165765-1852-47c4-af66-1d2425d86373' 2024-09-15T03:51:18.278 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:51:18.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:51:18.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC2WeZmo8x4ERAA2Upa7zjzt0T1obzUx6TG1w== 2024-09-15T03:51:18.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC2WeZmo8x4ERAA2Upa7zjzt0T1obzUx6TG1w=="}' 2024-09-15T03:51:18.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3d165765-1852-47c4-af66-1d2425d86373 -i td/divergent-priors/2/new.json 2024-09-15T03:51:18.796 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:51:18.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T03:51:18.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2WeZmo8x4ERAA2Upa7zjzt0T1obzUx6TG1w== --osd-uuid 3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:51:18.839 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.838+0000 7f634e61a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:18.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.840+0000 7f634e61a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:18.842 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.841+0000 7f634e61a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:18.842 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.842+0000 7f634e61a440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:51:18.842 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.842+0000 7f634e61a440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:51:18.842 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.842+0000 7f634e61a440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:51:18.920 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:18.919+0000 7f634e61a440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T03:51:25.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T03:51:25.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:51:25.114 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T03:51:25.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:51:25.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:51:25.604 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:51:25.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:51:25.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:51:25.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:51:25.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:51:25.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:51:25.634 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:25.634+0000 7f0a7560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:25.654+0000 7f0a7560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:25.656 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:25.655+0000 7f0a7560c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:51:26.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:26.001 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:51:26.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:51:26.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:26.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:26.183 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:26.183+0000 7f0a7560c440 -1 Falling back to public interface 2024-09-15T03:51:26.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:27.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:27.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:27.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:51:27.412 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:51:27.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:27.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:27.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:28.761 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:51:28.758+0000 7f0a7560c440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:51:28.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:28.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:28.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:51:28.814 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:51:28.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:28.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:29.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:30.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:30.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:30.447 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:51:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:51:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:30.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:30.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:31.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:31.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:31.829 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:51:31.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:51:31.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:31.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:32.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:33.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:33.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:33.291 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:51:33.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:51:33.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:33.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:33.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:34.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:34.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:34.754 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:51:34.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:51:34.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:34.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:35.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:36.159 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:51:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:51:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:37.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:37.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:37.569 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:51:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:51:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:37.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:38.933 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:51:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:51:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:39.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:40.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:40.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:40.300 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:51:40.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:51:40.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:40.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:40.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:41.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:41.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:41.697 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:51:41.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:51:41.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:41.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:42.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:43.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:43.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:43.208 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:51:43.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:51:43.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:43.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:43.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:44.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:44.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:44.761 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:51:44.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:51:44.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:44.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:45.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:46.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:46.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:46.564 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:51:46.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:51:46.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:46.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:47.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:48.569 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:51:48.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:51:48.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:48.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:49.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:50.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:50.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:50.292 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:51:50.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:51:50.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:50.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:51.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:51:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:51:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:51:52.110 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:51:52.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:51:52.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:51:52.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:51:52.478 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/64292,v1:127.0.0.1:6819/64292] [v2:127.0.0.1:6820/64292,v1:127.0.0.1:6821/64292] exists,up 3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:51:52.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:51:52.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:51:52.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:51:52.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-09-15T03:51:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T03:51:53.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-09-15T03:51:54.159 INFO:tasks.workunit.client.0.smithi138.stderr:noin is set 2024-09-15T03:51:54.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-09-15T03:51:55.089 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T03:51:55.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-09-15T03:51:55.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T03:51:55.945 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T03:51:55.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:51:56.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-09-15T03:51:57.618 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T03:51:57.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-09-15T03:51:58.507 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T03:51:58.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-09-15T03:51:59.202 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 pg_autoscale_mode to on 2024-09-15T03:51:59.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-09-15T03:51:59.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1726372319 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:51:59.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:51:59.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:51:59.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T03:51:59.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T03:51:59.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T03:51:59.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:51:59.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:51:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-15T03:51:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-15T03:51:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672975' 2024-09-15T03:51:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:51:59.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:52:00.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-15T03:52:00.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-15T03:52:00.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672975 2-64424509448' 2024-09-15T03:52:00.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:00.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T03:52:00.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:00.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:52:00.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T03:52:00.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:00.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T03:52:00.046 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836500 2024-09-15T03:52:00.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T03:52:00.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:52:00.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836500 2024-09-15T03:52:00.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:52:01.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:52:01.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:52:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T03:52:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:01.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-15T03:52:01.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:52:01.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-15T03:52:01.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:01.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-15T03:52:01.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T03:52:01.764 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672975 2024-09-15T03:52:01.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:52:02.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T03:52:02.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:02.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-15T03:52:02.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:02.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:52:02.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-15T03:52:02.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:02.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-15T03:52:02.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T03:52:02.143 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509448 2024-09-15T03:52:02.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:52:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:52:02.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:52:02.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:52:02.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:52:02.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:52:02.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:52:02.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:52:02.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:52:02.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:52:02.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:52:03.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:52:03.001 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:52:03.002 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:52:03.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:52:03.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:03.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:52:03.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-15T03:52:03.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-15T03:52:03.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-15T03:52:03.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:03.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:52:03.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-15T03:52:03.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-15T03:52:03.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672977' 2024-09-15T03:52:03.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:03.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509451 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509451 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672977 2-64424509451' 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-15T03:52:03.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:03.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:52:03.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-15T03:52:03.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:03.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-15T03:52:03.353 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836502 2024-09-15T03:52:03.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-15T03:52:03.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:52:03.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-15T03:52:03.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-15T03:52:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:03.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:52:03.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-15T03:52:03.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:03.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-15T03:52:03.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T03:52:03.707 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672977 2024-09-15T03:52:03.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:52:04.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672977 2024-09-15T03:52:04.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:04.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509451 2024-09-15T03:52:04.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:04.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:52:04.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509451 2024-09-15T03:52:04.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:04.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509451 2024-09-15T03:52:04.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-09-15T03:52:04.082 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509451 2024-09-15T03:52:04.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:04.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509451 2024-09-15T03:52:04.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:52:05.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:52:05.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:05.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509451 -lt 64424509451 2024-09-15T03:52:05.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:52:05.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:05.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:06.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:52:06.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:52:06.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:52:06.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:52:06.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:52:06.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:52:06.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:52:06.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:52:06.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:52:06.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:52:06.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:06.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:07.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:52:07.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:52:07.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:52:07.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-09-15T03:52:07.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-09-15T03:52:07.479 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:52:07.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-09-15T03:52:07.489 INFO:tasks.workunit.client.0.smithi138.stdout:primary and soon to be divergent is 1 2024-09-15T03:52:07.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-09-15T03:52:07.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:52:07.821 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:52:07.822 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T03:51:56.431448+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:03:43.876543+0000 0 0 2024-09-15T03:52:07.822 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:52:07.822 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:52:07.823 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:52:07.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-09-15T03:52:07.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1726372327 2024-09-15T03:52:07.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1726372327 - 1726372319 2024-09-15T03:52:07.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=8 2024-09-15T03:52:07.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 8 -gt 300 ']' 2024-09-15T03:52:07.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-09-15T03:52:07.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-09-15T03:52:07.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-09-15T03:52:07.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-09-15T03:52:07.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-09-15T03:52:07.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-09-15T03:52:07.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-09-15T03:52:07.841 INFO:tasks.workunit.client.0.smithi138.stdout:writing initial objects 2024-09-15T03:52:07.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-09-15T03:52:07.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:07.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-09-15T03:52:07.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:07.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-09-15T03:52:08.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-09-15T03:52:08.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-09-15T03:52:08.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-09-15T03:52:08.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-09-15T03:52:08.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-09-15T03:52:08.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-09-15T03:52:08.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:08.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-09-15T03:52:09.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-09-15T03:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-09-15T03:52:09.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-09-15T03:52:09.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-09-15T03:52:09.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-09-15T03:52:09.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-09-15T03:52:09.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:09.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-09-15T03:52:10.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-09-15T03:52:10.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-09-15T03:52:10.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-09-15T03:52:10.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-09-15T03:52:10.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-09-15T03:52:10.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-09-15T03:52:10.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:10.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-09-15T03:52:11.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-09-15T03:52:11.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-09-15T03:52:11.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-09-15T03:52:11.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-09-15T03:52:11.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-09-15T03:52:11.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:11.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-09-15T03:52:12.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-09-15T03:52:12.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-09-15T03:52:12.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-09-15T03:52:12.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-09-15T03:52:12.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-09-15T03:52:12.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-09-15T03:52:12.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-09-15T03:52:12.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:12.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-09-15T03:52:13.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-09-15T03:52:13.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-09-15T03:52:13.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-09-15T03:52:13.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-09-15T03:52:13.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-09-15T03:52:13.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-09-15T03:52:13.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:13.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-09-15T03:52:14.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-09-15T03:52:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-09-15T03:52:14.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-09-15T03:52:14.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-09-15T03:52:14.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-09-15T03:52:14.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:14.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-09-15T03:52:15.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:15.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-09-15T03:52:15.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:15.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-09-15T03:52:15.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:15.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-09-15T03:52:15.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:15.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-09-15T03:52:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-09-15T03:52:16.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-09-15T03:52:16.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-09-15T03:52:16.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-09-15T03:52:16.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-09-15T03:52:16.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-09-15T03:52:16.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-09-15T03:52:16.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:16.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-09-15T03:52:17.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:17.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-09-15T03:52:17.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:17.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-09-15T03:52:17.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:17.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-09-15T03:52:17.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:17.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-09-15T03:52:18.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:18.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-09-15T03:52:18.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:18.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-09-15T03:52:19.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:19.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-09-15T03:52:19.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:19.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-09-15T03:52:19.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:19.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-09-15T03:52:19.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:19.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-09-15T03:52:19.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:19.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-09-15T03:52:20.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-09-15T03:52:20.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-09-15T03:52:20.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-09-15T03:52:20.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-09-15T03:52:20.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-09-15T03:52:20.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:20.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-09-15T03:52:21.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-09-15T03:52:21.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-09-15T03:52:21.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-09-15T03:52:21.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-09-15T03:52:21.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-09-15T03:52:21.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:21.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-09-15T03:52:22.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-09-15T03:52:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-09-15T03:52:22.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-09-15T03:52:22.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-09-15T03:52:22.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-09-15T03:52:22.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:22.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-09-15T03:52:23.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-09-15T03:52:23.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-09-15T03:52:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-09-15T03:52:23.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-09-15T03:52:23.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-09-15T03:52:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-09-15T03:52:23.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:23.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-09-15T03:52:24.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:24.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-09-15T03:52:24.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-09-15T03:52:24.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:52:24.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:52:24.435 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:52:24.435 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:52:24.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:52:24.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:52:24.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:52:24.489 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:24.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:52:24.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T03:52:24.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T03:52:24.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T03:52:24.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:24.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:52:25.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672984 2024-09-15T03:52:25.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672984 2024-09-15T03:52:25.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672984' 2024-09-15T03:52:25.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:25.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:52:25.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509457 2024-09-15T03:52:25.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509457 2024-09-15T03:52:25.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672984 2-64424509457' 2024-09-15T03:52:25.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:25.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T03:52:25.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:25.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:52:25.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T03:52:25.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:25.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T03:52:25.230 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836509 2024-09-15T03:52:25.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T03:52:25.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:52:25.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T03:52:25.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:25.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672984 2024-09-15T03:52:25.714 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:25.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:52:25.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672984 2024-09-15T03:52:25.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:25.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672984 2024-09-15T03:52:25.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-09-15T03:52:25.716 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672984 2024-09-15T03:52:25.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:52:26.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672984 -lt 42949672984 2024-09-15T03:52:26.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:26.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509457 2024-09-15T03:52:26.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:26.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:52:26.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509457 2024-09-15T03:52:26.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:26.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509457 2024-09-15T03:52:26.263 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509457 2024-09-15T03:52:26.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-09-15T03:52:26.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509457 2024-09-15T03:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:27.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T03:52:27.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:52:27.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:52:27.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:52:27.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:52:27.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:52:27.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:52:27.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:52:27.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-15T03:52:27.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:52:27.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:27.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:28.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-15T03:52:28.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-15T03:52:28.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T03:52:28.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-09-15T03:52:28.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-09-15T03:52:28.690 INFO:tasks.workunit.client.0.smithi138.stdout:blackholing osds 0 2 2024-09-15T03:52:28.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:26.527487+0000 34'66 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:18.605572+0000 34'66 34:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.527162+0000 34'67 36:104 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:26.527333+0000 34'66 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:18.604771+0000 32'65 34:91 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526973+0000 34'67 36:104 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:18.599708+0000 34'67 34:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:26.527172+0000 32'65 36:90 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:18.603350+0000 34'67 34:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526731+0000 34'67 36:104 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 966 0 0 17 48 17 active+clean 2024-09-15T03:52:24.214813+0000 32'65 34:101 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:56:52.048643+0000 2 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 active+clean 2024-09-15T03:52:26.526629+0000 34'66 36:113 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:32:01.142291+0000 4 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 1 0 0 0 0 483 0 0 18 48 18 active+clean 2024-09-15T03:52:22.187394+0000 34'66 34:102 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:29:06.395353+0000 1 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:18.603789+0000 34'66 34:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526400+0000 34'67 36:113 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:11:04.608308+0000 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:18.606948+0000 32'65 34:91 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.033 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:52:19.286148+0000 34'66 34:103 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:23:46.730759+0000 2 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.527001+0000 34'67 36:104 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:18.606280+0000 34'67 34:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:26.526549+0000 34'66 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:18.604951+0000 34'66 34:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:26.527004+0000 32'65 36:90 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:25.230510+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:18.606431+0000 32'65 34:91 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526761+0000 34'67 36:104 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:18.605378+0000 34'67 34:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:26.527148+0000 34'66 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:18.604261+0000 34'66 34:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.034 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:26.526810+0000 32'65 36:90 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:18.606710+0000 34'67 34:93 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:26.526908+0000 32'65 36:90 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:18.605693+0000 34'66 34:92 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:26.527361+0000 34'66 36:103 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:52:29.035 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:52:29.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:29.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-09-15T03:52:29.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:52:29.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:52:29.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T03:52:29.116 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:52:29.116 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:52:29.116 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:52:29.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:52:29.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:52:29.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:52:29.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T03:52:29.196 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:52:29.196 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:52:29.197 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:52:29.203 INFO:tasks.workunit.client.0.smithi138.stdout:create new divergent objects 2024-09-15T03:52:29.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-09-15T03:52:29.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-09-15T03:52:29.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:52:29.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:52:29.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-09-15T03:52:29.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-09-15T03:52:29.205 INFO:tasks.workunit.client.0.smithi138.stdout:writing divergent objects 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-09-15T03:52:29.206 INFO:tasks.workunit.client.0.smithi138.stdout:remove divergent objects 2024-09-15T03:52:29.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-09-15T03:52:29.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-09-15T03:52:29.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-09-15T03:52:29.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T03:52:29.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-09-15T03:52:29.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-09-15T03:52:29.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-09-15T03:52:29.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-09-15T03:52:29.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-09-15T03:52:29.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-09-15T03:52:29.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-09-15T03:52:29.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-09-15T03:52:39.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-09-15T03:52:39.219 INFO:tasks.workunit.client.0.smithi138.stderr:rados: no process found 2024-09-15T03:52:39.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-09-15T03:52:39.220 INFO:tasks.workunit.client.0.smithi138.stdout:killing all the osds 2024-09-15T03:52:39.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:52:39.574 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:52:39.574 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:30.095482+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.574 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:35.926555+0000 34'66 44:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.574 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.527162+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:30.011021+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 3 0 0 0 0 1449 0 0 17 48 17 active+clean 2024-09-15T03:52:37.132592+0000 32'65 44:107 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:37.132479+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:26:10.215143+0000 3 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 0 0 0 1618 0 0 20 48 20 active+clean 2024-09-15T03:52:26.526973+0000 38'68 43:113 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 3 0 0 0 0 1449 0 0 19 48 19 active+clean 2024-09-15T03:52:26.370997+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:52:10.194430+0000 3 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:30.095684+0000 32'65 44:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 0 0 0 1932 0 0 19 48 19 active+clean 2024-09-15T03:52:35.985106+0000 34'67 44:118 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T13:36:15.741471+0000 4 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526731+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 966 0 0 17 48 17 active+clean 2024-09-15T03:52:24.214813+0000 32'65 44:112 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:56:52.048643+0000 2 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 active+clean 2024-09-15T03:52:30.011341+0000 34'66 44:119 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:10:06.826580+0000 4 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 1 0 0 0 0 483 0 0 18 48 18 active+clean 2024-09-15T03:52:34.112614+0000 34'66 44:120 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:34.112510+0000 34'66 2024-09-15T03:52:22.187335+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:12:24.993681+0000 1 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:52:28.319794+0000 34'66 44:113 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:59:17.460537+0000 3 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 active+clean 2024-09-15T03:52:26.526400+0000 34'67 43:121 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:11:04.608308+0000 0 0 2024-09-15T03:52:39.575 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 966 0 0 17 48 17 active+clean 2024-09-15T03:52:30.221178+0000 32'65 44:112 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:30.221117+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T15:41:33.739021+0000 2 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:52:19.286148+0000 34'66 44:114 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:23:46.730759+0000 2 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:26.527001+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 0 0 0 1464 0 0 21 48 21 active+clean 2024-09-15T03:52:35.926894+0000 38'69 44:117 [1,0,2] 1 [1,0,2] 1 38'69 2024-09-15T03:52:35.097400+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:30.095664+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:52:36.285657+0000 34'66 44:113 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:36.285595+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T11:25:31.304252+0000 3 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 966 0 0 18 48 18 active+clean 2024-09-15T03:52:30.778316+0000 38'66 43:109 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 1 periodic scrub scheduled @ 2024-09-16T11:00:21.061864+0000 2 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 1 0 0 0 0 483 0 0 17 48 17 active+clean 2024-09-15T03:52:29.101410+0000 32'65 44:107 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:14:05.539642+0000 1 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:52:30.094957+0000 34'67 44:110 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 3 0 0 0 0 1449 0 0 19 48 19 active+clean 2024-09-15T03:52:34.259201+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:34.259105+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:53:40.086253+0000 3 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:52:31.804093+0000 34'66 43:121 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:31.803983+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T15:07:33.424311+0000 3 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:35.926692+0000 34'66 44:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.576 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:30.095266+0000 32'65 44:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 3 0 0 0 0 1449 0 0 19 48 19 active+clean 2024-09-15T03:52:32.279679+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:32.279615+0000 34'67 2024-09-15T03:52:32.279615+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:09:02.892471+0000 3 0 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:52:26.526908+0000 32'65 43:98 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:52:28.512723+0000 34'66 44:110 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 4 0 0 0 0 1135 0 0 20 48 20 active+clean 2024-09-15T03:52:30.095909+0000 38'68 44:122 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:52:39.577 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:52:39.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-09-15T03:52:39.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:52:39.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:52:39.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:52:39.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:52:39.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:52:39.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:52:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T03:52:39.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-09-15T03:52:40.275 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T03:52:40.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T03:52:40.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-09-15T03:52:40.779 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T03:52:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-09-15T03:52:40.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-09-15T03:52:41.322 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T03:52:41.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:41.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-09-15T03:52:41.849 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T03:52:41.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:41.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-09-15T03:52:42.350 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T03:52:42.361 INFO:tasks.workunit.client.0.smithi138.stdout:bringing up non_divergent 0 2 2024-09-15T03:52:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-09-15T03:52:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:30.095482+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:35.926555+0000 34'66 44:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:26.527162+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:30.011021+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 3 0 0 0 0 1449 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:37.132592+0000 32'65 44:107 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:37.132479+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:26:10.215143+0000 3 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 0 0 0 1618 0 0 20 48 20 stale+active+clean 2024-09-15T03:52:26.526973+0000 38'68 43:113 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 3 0 0 0 0 1449 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:26.370997+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:52:10.194430+0000 3 0 2024-09-15T03:52:42.732 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:30.095684+0000 32'65 44:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 0 0 0 1932 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:35.985106+0000 34'67 44:118 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T13:36:15.741471+0000 4 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:26.526731+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 966 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:24.214813+0000 32'65 44:112 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:56:52.048643+0000 2 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:30.011341+0000 34'66 44:119 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:10:06.826580+0000 4 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 1 0 0 0 0 483 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:34.112614+0000 34'66 44:120 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:34.112510+0000 34'66 2024-09-15T03:52:22.187335+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:12:24.993681+0000 1 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1449 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:28.319794+0000 34'66 44:113 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:59:17.460537+0000 3 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:26.526400+0000 34'67 43:121 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:11:04.608308+0000 0 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 966 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:30.221178+0000 32'65 44:112 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:30.221117+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T15:41:33.739021+0000 2 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:19.286148+0000 34'66 44:114 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:23:46.730759+0000 2 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:26.527001+0000 34'67 43:112 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 0 0 0 1464 0 0 21 48 21 stale+active+clean 2024-09-15T03:52:35.926894+0000 38'69 44:117 [1,0,2] 1 [1,0,2] 1 38'69 2024-09-15T03:52:35.097400+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.733 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:30.095664+0000 34'66 44:109 [1,2,0] 1 [1,2,0] 1 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1449 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:36.285657+0000 34'66 44:113 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:36.285595+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T11:25:31.304252+0000 3 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 966 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:30.778316+0000 38'66 43:109 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 1 periodic scrub scheduled @ 2024-09-16T11:00:21.061864+0000 2 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 1 0 0 0 0 483 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:29.101410+0000 32'65 44:107 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:14:05.539642+0000 1 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:30.094957+0000 34'67 44:110 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 3 0 0 0 0 1449 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:34.259201+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:34.259105+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:53:40.086253+0000 3 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1449 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:31.804093+0000 34'66 43:121 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:31.803983+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T15:07:33.424311+0000 3 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:35.926692+0000 34'66 44:107 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:30.095266+0000 32'65 44:108 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 3 0 0 0 0 1449 0 0 19 48 19 stale+active+clean 2024-09-15T03:52:32.279679+0000 34'67 44:114 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:32.279615+0000 34'67 2024-09-15T03:52:32.279615+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:09:02.892471+0000 3 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 stale+active+clean 2024-09-15T03:52:26.526908+0000 32'65 43:98 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 stale+active+clean 2024-09-15T03:52:28.512723+0000 34'66 44:110 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 4 0 0 0 0 1135 0 0 20 48 20 stale+active+clean 2024-09-15T03:52:30.095909+0000 38'68 44:122 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:52:42.734 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:52:42.735 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:52:42.735 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:52:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-09-15T03:52:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:52:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:52:42.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:52:42.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:52:42.749 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:52:42.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:52:42.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:52:42.752 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:52:42.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:52:42.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:52:42.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T03:52:42.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:52:42.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:52:42.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:52:42.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:52:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:42.773+0000 7f0473380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:42.776 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:42.776+0000 7f0473380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:42.778 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:42.777+0000 7f0473380440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:43.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:52:43.110 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:52:43.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:43.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:52:43.290 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:43.290+0000 7f0473380440 -1 Falling back to public interface 2024-09-15T03:52:43.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:44.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:44.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:44.494 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:52:44.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:52:44.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:44.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:52:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:52:45.860 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:52:45.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:45.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:52:46.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:46.853 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:46.853+0000 7f0473380440 -1 osd.0 38 log_to_monitors true 2024-09-15T03:52:47.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:47.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:47.274 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:52:47.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:52:47.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:47.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:52:47.707 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 57 up_thru 37 down_at 48 last_clean_interval [5,38) [v2:127.0.0.1:6802/68618,v1:127.0.0.1:6803/68618] [v2:127.0.0.1:6804/68618,v1:127.0.0.1:6805/68618] exists,up 516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:52:47.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:52:47.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:52:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:52:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:52:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:52:47.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:52:47.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:52:47.714 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:52:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:52:47.739 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:47.738+0000 7f4c4b809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:47.741 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:47.740+0000 7f4c4b809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:47.742 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:47.742+0000 7f4c4b809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:52:48.069 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:52:48.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:48.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:52:48.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:48.258+0000 7f4c4b809440 -1 Falling back to public interface 2024-09-15T03:52:48.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:49.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:49.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:49.425 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:52:49.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:52:49.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:49.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:52:49.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:50.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:50.817 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:52:50.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:50.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:52:50.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:50.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:52:51.227 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:52:51.226+0000 7f4c4b809440 -1 osd.2 38 log_to_monitors true 2024-09-15T03:52:51.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:52:52.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:52:52.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:52:52.367 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:52:52.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:52:52.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:52:52.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:52:52.721 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up out weight 0 up_from 61 up_thru 35 down_at 50 last_clean_interval [15,38) [v2:127.0.0.1:6810/69196,v1:127.0.0.1:6811/69196] [v2:127.0.0.1:6812/69196,v1:127.0.0.1:6813/69196] exists,up 3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:52:52.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:52:52.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:52:52.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:52:52.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:52.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-09-15T03:52:53.300 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T03:52:53.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:52:53.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-09-15T03:52:54.326 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already in. 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:52:54.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:52:54.338 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:52:54.338 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:52:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:52:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:52:54.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:52:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:52:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:52:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:52:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:52:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:52:54.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:52:54.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:52:54.905 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:52:54.905 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:52:54.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:52:54.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:54.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:52:55.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135877 2024-09-15T03:52:55.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135877 2024-09-15T03:52:55.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135877' 2024-09-15T03:52:55.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:55.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:52:55.225 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:52:55.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:52:55.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:52:55.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:52:55.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:52:55.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:52:55.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=261993005060 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 261993005060 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135877 2-261993005060' 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-244813135877 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:55.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:52:55.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-244813135877 2024-09-15T03:52:55.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:55.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135877 2024-09-15T03:52:55.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 244813135877' 2024-09-15T03:52:55.344 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 244813135877 2024-09-15T03:52:55.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:52:55.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135877 -lt 244813135877 2024-09-15T03:52:55.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:52:55.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-261993005060 2024-09-15T03:52:55.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:52:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:52:55.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-261993005060 2024-09-15T03:52:55.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:52:55.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=261993005060 2024-09-15T03:52:55.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 261993005060' 2024-09-15T03:52:55.727 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 261993005060 2024-09-15T03:52:55.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:56.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005058 -lt 261993005060 2024-09-15T03:52:56.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:52:57.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:52:57.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:52:57.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005060 -lt 261993005060 2024-09-15T03:52:57.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:52:57.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:57.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:52:58.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:52:58.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:52:58.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:52:58.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:58.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:52:58.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:52:58.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:52:59.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:52:59.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:52:59.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:52:59.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:52:59.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:52:59.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:00.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:00.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:00.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:53:00.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:00.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:53:00.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:00.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:00.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:00.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:00.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:00.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:01.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:01.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:01.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:01.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:01.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:01.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:53:01.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:01.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:02.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:02.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:02.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:02.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:02.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:02.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:02.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:02.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:03.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:03.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:03.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:53:03.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:03.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:53:04.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:04.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:04.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:04.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:04.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:04.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:04.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:04.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:04.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:04.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:04.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:04.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:05.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:05.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:05.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:05.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:53:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:53:07.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:07.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:07.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:07.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:07.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:07.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:07.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:07.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:07.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:07.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:07.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:07.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:08.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:08.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:08.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:08.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:08.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:08.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:53:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:53:11.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:11.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:11.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:12.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:12.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:12.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:12.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:12.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:12.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:12.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:12.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:13.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:13.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:13.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:53:13.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:13.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:53:19.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:19.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:19.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:19.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:19.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:19.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:19.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:19.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:19.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:19.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:19.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:19.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:20.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:20.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:20.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:20.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:20.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:20.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:20.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:53:20.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:20.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:53:28.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:28.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:28.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:28.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:28.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:28.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:28.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:28.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:28.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:28.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:29.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:29.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:29.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:53:29.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:53:29.965 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:53:29.965 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "05b9a170-bdb5-4e05-9af6-dbc6a14be80a", 2024-09-15T03:53:29.965 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:53:29.965 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:53:29.964451+0000", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:53:29.966 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.967 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 32 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.969 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.975 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [2,0]" 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:53:29.976 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.977 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:50:10.137844Z", 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:50:10.137844Z", 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:53:29.978 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:53:29.979 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:29.980 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:53:29.981 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.986 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 67, 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:50:11.009609+0000", 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:52:54.676027+0000", 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:52:51.664326+0000", 2024-09-15T03:53:29.987 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:52:53.838276+0000", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:53:29.988 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:51:55.401374+0000", 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T03:53:29.989 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "on", 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 32, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 16, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 32, 2024-09-15T03:53:29.990 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 32, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 32, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "63", 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:53:29.991 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "33", 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:53:29.992 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:53:29.993 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:53:29.994 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "516e25a2-58bf-4649-896b-6cdc14377ee5", 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:53:29.995 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 38, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 57, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 65, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 48, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.996 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:53:29.997 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:53:29.998 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:29.999 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.000 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/68618", 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/68618", 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/68618", 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/68618", 2024-09-15T03:53:30.001 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "dd75342f-e3b1-4fab-8134-909af98dedc4", 2024-09-15T03:53:30.002 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 46, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 49, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:53:30.003 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.004 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.005 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/62563", 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/62563", 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62563", 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62563", 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.009 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "3d165765-1852-47c4-af66-1d2425d86373", 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 38, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 61, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 66, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 50, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.011 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:53:30.012 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:53:30.013 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.014 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/69196", 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/69196", 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/69196", 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/69196", 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:53:30.015 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:40.027686+0000", 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:50:43.603685+0000", 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:40.593560+0000", 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:53:30.016 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:51:17.698525+0000", 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:41.139433+0000", 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:51:51.529175+0000", 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:53:30.017 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:53:30.018 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.019 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/68618,v1:127.0.0.1:6805/68618]", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:53:30.020 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:50:16.850812Z", 2024-09-15T03:53:30.021 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/68618,v1:127.0.0.1:6803/68618]", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/68618,v1:127.0.0.1:6809/68618]", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/68618,v1:127.0.0.1:6807/68618]", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:53:30.022 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.023 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/62563,v1:127.0.0.1:6813/62563]", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:53:30.024 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:53:30.025 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:50:50.789574Z", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/62563,v1:127.0.0.1:6811/62563]", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:53:30.026 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62563,v1:127.0.0.1:6817/62563]", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62563,v1:127.0.0.1:6815/62563]", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:53:30.027 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/69196,v1:127.0.0.1:6813/69196]", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:53:30.028 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.029 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:51:24.443855Z", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/69196,v1:127.0.0.1:6811/69196]", 2024-09-15T03:53:30.030 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/69196,v1:127.0.0.1:6817/69196]", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/69196,v1:127.0.0.1:6815/69196]", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:53:30.031 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [ 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.032 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "floor": 36 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 58 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 61 2024-09-15T03:53:30.033 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 67, 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:53:30.034 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:53:30.035 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.036 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:53:30.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:53:30.038 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.039 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.040 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:53:30.041 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:53:30.042 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:53:30.043 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:53:30.043 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.043 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:53:30.043 INFO:tasks.workunit.client.0.smithi138.stderr:report 1962098971 2024-09-15T03:53:30.043 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:53:30.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:53:30.045 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:53:30.046 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.047 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:53:30.048 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:53:30.049 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:53:30.050 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:53:30.051 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.052 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:53:30.053 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 32, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 32, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48721, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:53:30.054 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 90, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 90, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:53:30.055 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:53:30.056 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.057 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 579, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 579, 2024-09-15T03:53:30.058 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 64, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 64, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 96, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895448, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2648, 2024-09-15T03:53:30.059 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892725, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677352, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205608448, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2711552, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 627656, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:53:30.060 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914150874 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 96, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 96, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 96000000, 2024-09-15T03:53:30.061 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 96000000 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 96, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:53:30.062 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895448, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2648, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892725, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677352, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205608448, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2711552, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 627656, 2024-09-15T03:53:30.063 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914150874 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:53:30.064 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 96, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 96, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 96000000, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 96000000 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:53:30.065 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 32, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48721, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:53:30.066 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 90, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 90, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:53:30.067 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:53:30.068 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1220608, 2024-09-15T03:53:30.069 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 143276, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 579, 2024-09-15T03:53:30.070 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 579, 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 64, 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 64, 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 244813135884 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.071 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672987 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 261993005067 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.072 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 32 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 20, 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 32, 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:53:30.073 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 12, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 32, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.074 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 5, 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:52:43.279278+0000", 2024-09-15T03:53:30.075 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 561, 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:53:30.076 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:53:30.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:53:30.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-09-15T03:53:30.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-09-15T03:53:30.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-09-15T03:53:30.078 INFO:tasks.workunit.client.0.smithi138.stdout:writing non-divergent object existing_4 2024-09-15T03:53:30.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:53:30.317 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:53:30.317 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051087+0000 34'66 67:115 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050082+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 67:90 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 67:104 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:126 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051096+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:101 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 67:125 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 3 0 0 1465 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048969+0000 34'66 67:125 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.318 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:114 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 2 0 0 966 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050921+0000 34'67 67:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:07:10.547853+0000 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:101 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:125 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049257+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 34'67 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 4 0 0 1932 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050712+0000 34'66 67:124 [2,0] 2 [2,0] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:24:11.573499+0000 4 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.049786+0000 32'65 67:104 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:25.230510+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050810+0000 32'65 67:112 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 67:116 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:09.986331+0000 34'66 67:117 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.319 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:53:30.320 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:53:30.321 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:53:30.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T03:53:30.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:53:30.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:53:30.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:53:30.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:53:30.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:53:30.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:53:30.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:53:30.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:53:30.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:53:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:53:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:53:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:53:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:53:30.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:53:30.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:53:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=244813135886 2024-09-15T03:53:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 244813135886 2024-09-15T03:53:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135886' 2024-09-15T03:53:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:53:30.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:53:31.031 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:53:31.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:53:31.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:53:31.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:53:31.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:53:31.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=261993005069 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 261993005069 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-244813135886 2-261993005069' 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-244813135886 2024-09-15T03:53:31.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:53:31.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:53:31.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-244813135886 2024-09-15T03:53:31.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:53:31.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=244813135886 2024-09-15T03:53:31.150 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 244813135886 2024-09-15T03:53:31.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 244813135886' 2024-09-15T03:53:31.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:53:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 244813135886 -lt 244813135886 2024-09-15T03:53:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:53:31.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-261993005069 2024-09-15T03:53:31.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:53:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:53:31.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-261993005069 2024-09-15T03:53:31.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:53:31.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=261993005069 2024-09-15T03:53:31.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 261993005069' 2024-09-15T03:53:31.525 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 261993005069 2024-09-15T03:53:31.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:53:31.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 261993005069 -lt 261993005069 2024-09-15T03:53:31.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:53:31.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:31.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:32.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:32.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:32.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:32.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:32.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:33.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:33.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:53:33.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:53:33.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:53:33.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:33.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:33.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:33.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:33.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:33.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:33.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:34.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:34.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:34.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:34.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:34.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:53:34.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:34.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:53:34.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:34.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:34.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:35.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:35.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:35.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:35.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:35.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:35.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:35.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:35.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:36.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:36.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:36.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:53:36.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:36.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:53:36.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:36.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:36.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:36.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:36.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:37.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:37.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:37.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:37.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:37.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:37.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:37.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:37.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:37.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:53:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:38.717 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:38.717 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:39.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:39.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:39.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:39.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:39.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:40.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:40.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:40.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:53:40.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:40.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:53:41.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:41.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:41.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:41.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:41.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:41.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:42.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:42.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:42.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:42.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:42.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:42.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:42.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:53:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:53:46.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:46.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:46.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:46.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:46.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:46.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:46.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:46.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:46.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:46.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:46.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:46.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:46.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:46.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:46.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:46.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:47.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:47.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:47.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:53:47.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:47.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:53:53.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:53:53.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:53:53.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:53:54.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:53:54.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:53:54.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:53:54.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:53:54.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:53:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:53:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:53:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:53:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:53:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:54:02.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:54:02.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:54:02.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:54:02.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:54:02.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:54:02.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:54:02.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:54:02.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:54:02.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:54:02.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:54:02.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:54:02.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:54:03.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:54:03.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:54:03.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:54:03.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:54:03.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:54:03.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:54:04.411 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "05b9a170-bdb5-4e05-9af6-dbc6a14be80a", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:54:04.410782+0000", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:54:04.412 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:54:04.413 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:54:04.413 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:54:04.413 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.416 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:54:04.416 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:54:04.416 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.416 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:54:04.416 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:54:04.417 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 64 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.418 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.2 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.3 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.4 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.5 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.419 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.6 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.7 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.8 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.9 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.420 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.a is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.b is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.c is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.d is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.e is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.f is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.10 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.11 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.12 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.422 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.13 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.14 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.15 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.16 is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.423 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.17 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.18 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.19 is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1a is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1b is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.424 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1c is stuck undersized for 68s, current state active+undersized+degraded, last acting [0,2]" 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1d is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1e is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.1f is stuck undersized for 68s, current state active+undersized+degraded, last acting [2,0]" 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:54:04.425 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.426 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:50:10.137844Z", 2024-09-15T03:54:04.427 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:50:10.137844Z", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:54:04.428 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.429 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.430 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 67, 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:50:11.009609+0000", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:52:54.676027+0000", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:52:51.664326+0000", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:52:53.838276+0000", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:54:04.431 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:54:04.432 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:51:55.401374+0000", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T03:54:04.433 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "on", 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 32, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 16, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 32, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 32, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 32, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:54:04.434 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "63", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "33", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T03:54:04.435 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:54:04.436 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T03:54:04.437 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "516e25a2-58bf-4649-896b-6cdc14377ee5", 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:54:04.438 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 38, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 57, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 65, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 48, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.439 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.440 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.441 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.442 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 68618 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/68618", 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/68618", 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/68618", 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/68618", 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:54:04.443 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "dd75342f-e3b1-4fab-8134-909af98dedc4", 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 46, 2024-09-15T03:54:04.444 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 49, 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.445 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.446 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.447 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.448 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 62563 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/62563", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/62563", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/62563", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/62563", 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.449 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "3d165765-1852-47c4-af66-1d2425d86373", 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 38, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 61, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 66, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 50, 2024-09-15T03:54:04.450 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.451 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.452 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.453 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 69196 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.454 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/69196", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/69196", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/69196", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/69196", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:40.027686+0000", 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:54:04.455 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:50:43.603685+0000", 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:40.593560+0000", 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:51:17.698525+0000", 2024-09-15T03:54:04.456 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:52:41.139433+0000", 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:51:51.529175+0000", 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:54:04.457 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:54:04.458 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.459 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/68618,v1:127.0.0.1:6805/68618]", 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:54:04.460 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:54:04.461 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:50:16.850812Z", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:54:04.462 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/68618,v1:127.0.0.1:6803/68618]", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/68618,v1:127.0.0.1:6809/68618]", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/68618,v1:127.0.0.1:6807/68618]", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:54:04.463 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/62563,v1:127.0.0.1:6813/62563]", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:54:04.464 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:54:04.465 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:50:50.789574Z", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/62563,v1:127.0.0.1:6811/62563]", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/62563,v1:127.0.0.1:6817/62563]", 2024-09-15T03:54:04.466 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/62563,v1:127.0.0.1:6815/62563]", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:54:04.467 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/69196,v1:127.0.0.1:6813/69196]", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:54:04.468 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:54:04.469 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:51:24.443855Z", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:54:04.470 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/69196,v1:127.0.0.1:6811/69196]", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/69196,v1:127.0.0.1:6817/69196]", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/69196,v1:127.0.0.1:6815/69196]", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:54:04.471 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [ 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: "floor": 36 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.472 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 58 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 61 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 67, 2024-09-15T03:54:04.473 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:54:04.474 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:54:04.475 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.476 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stderr:report 2896427386 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.477 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:54:04.478 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:54:04.479 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:54:04.480 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:54:04.481 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.482 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:54:04.483 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.484 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.485 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.486 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:54:04.487 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:54:04.487 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:54:04.487 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:54:04.487 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:54:04.488 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:54:04.489 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:54:04.490 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 32, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 32, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:54:04.491 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48392, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:54:04.492 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 91, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 91, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:54:04.493 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:54:04.494 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 580, 2024-09-15T03:54:04.495 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 580, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 64, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 64, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 96, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895448, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2648, 2024-09-15T03:54:04.496 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892725, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677352, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205608448, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2711552, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 626998, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914150874 2024-09-15T03:54:04.497 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 96, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 96, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 96000000, 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 96000000 2024-09-15T03:54:04.498 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 96, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895448, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2648, 2024-09-15T03:54:04.499 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892725, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677352, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205608448, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2711552, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 626998, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914150874 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.500 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 96, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 96, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 96000000, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 96000000 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.501 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 32, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48392, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 100, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 300, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:54:04.502 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 100, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 100, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 91, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 91, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:54:04.503 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:54:04.504 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1220608, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 142618, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:54:04.505 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 580, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 580, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 64, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 64, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 244813135893 2024-09-15T03:54:04.506 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672987 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 261993005076 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 32 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.507 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 20, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 32, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 12, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 32, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.508 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:54:04.509 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 7, 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:53:53.295229+0000", 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 640, 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:54:04.510 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:54:04.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:54:04.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-09-15T03:54:04.511 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery 2024-09-15T03:54:04.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051087+0000 34'66 67:115 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050082+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 67:90 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 67:104 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.760 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:126 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051096+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:101 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 67:125 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 3 0 0 1465 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048969+0000 34'66 67:125 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:114 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 2 0 0 966 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050921+0000 34'67 67:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:07:10.547853+0000 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:101 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:125 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049257+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 67:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 4 0 0 1932 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050712+0000 34'66 67:124 [2,0] 2 [2,0] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:24:11.573499+0000 4 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.761 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.049786+0000 32'65 67:104 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:25.230510+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050810+0000 32'65 67:112 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 67:116 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:09.986331+0000 34'66 67:117 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:04.762 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:04.763 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:04.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:54:04.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-09-15T03:54:04.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:54:04.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:54:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:04.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:54:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:54:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T03:54:04.846 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:54:04.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:04.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:54:04.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:54:04.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T03:54:04.930 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:54:04.936 INFO:tasks.workunit.client.0.smithi138.stdout:revive divergent 1 2024-09-15T03:54:04.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-09-15T03:54:04.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051087+0000 34'66 67:115 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050082+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.273 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 67:90 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 67:104 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:126 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051096+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:101 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 67:125 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 3 0 0 1465 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048969+0000 34'66 67:125 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:114 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 2 0 0 966 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050921+0000 34'67 67:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:07:10.547853+0000 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:101 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.274 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:125 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049257+0000 34'67 67:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 67:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 4 0 0 1932 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050712+0000 34'66 67:124 [2,0] 2 [2,0] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:24:11.573499+0000 4 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.049786+0000 32'65 67:104 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:25.230510+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050810+0000 32'65 67:112 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 67:116 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:09.986331+0000 34'66 67:117 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:125 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 67:102 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.275 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 67:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.276 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 67:103 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.276 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:114 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:05.276 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:05.276 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:05.276 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:05.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-09-15T03:54:05.938 INFO:tasks.workunit.client.0.smithi138.stderr:noup is set 2024-09-15T03:54:05.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:54:05.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:05.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:54:05.955 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:54:05.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:54:05.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:54:05.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:54:05.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:54:05.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:54:05.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:54:05.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:54:05.982 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:05.980+0000 7fedfd11f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:05.984 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:05.982+0000 7fedfd11f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:05.985 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:05.983+0000 7fedfd11f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:06.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-15T03:54:06.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-09-15T03:54:06.498 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:06.497+0000 7fedfd11f440 -1 Falling back to public interface 2024-09-15T03:54:09.497 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:09.496+0000 7fedfd11f440 -1 osd.1 46 log_to_monitors true 2024-09-15T03:54:11.370 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery divergent 2024-09-15T03:54:11.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-09-15T03:54:11.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:11.716 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051087+0000 34'66 68:117 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050082+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 68:92 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 68:106 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:127 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051096+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:102 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.717 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 68:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 3 0 0 1465 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048969+0000 34'66 67:126 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:115 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 2 0 0 966 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.050921+0000 34'67 68:129 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:07:10.547853+0000 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:102 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:126 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049257+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 68:107 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 4 0 0 1932 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050712+0000 34'66 68:126 [2,0] 2 [2,0] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:24:11.573499+0000 4 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.049786+0000 32'65 68:106 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:25.230510+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050810+0000 32'65 67:113 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 68:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:116 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.718 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:09.986331+0000 34'66 68:119 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:11.719 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:54:11.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:54:11.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T03:54:11.803 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:54:11.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-09-15T03:54:12.438 INFO:tasks.workunit.client.0.smithi138.stderr:noup is unset 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:12.450 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:12.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:12.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:12.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:54:12.901 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 57 up_thru 65 down_at 48 last_clean_interval [5,38) [v2:127.0.0.1:6802/68618,v1:127.0.0.1:6803/68618] [v2:127.0.0.1:6804/68618,v1:127.0.0.1:6805/68618] exists,up 516e25a2-58bf-4649-896b-6cdc14377ee5 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:12.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:12.903 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:12.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:12.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:12.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:54:13.539 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 49 last_clean_interval [10,46) [v2:127.0.0.1:6818/71735,v1:127.0.0.1:6819/71735] [v2:127.0.0.1:6820/71735,v1:127.0.0.1:6821/71735] exists,up dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:54:13.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:13.542 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:13.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:13.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:54:13.922 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 61 up_thru 72 down_at 50 last_clean_interval [15,38) [v2:127.0.0.1:6810/69196,v1:127.0.0.1:6811/69196] [v2:127.0.0.1:6812/69196,v1:127.0.0.1:6813/69196] exists,up 3d165765-1852-47c4-af66-1d2425d86373 2024-09-15T03:54:13.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:54:13.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:54:13.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:54:13.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:14.289 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:14.289 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.512270+0000 34'66 72:122 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.289 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.289 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.508138+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 68:92 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 68:106 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:127 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.510788+0000 34'67 72:125 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:102 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 68:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 peering 2024-09-15T03:54:12.607015+0000 34'66 72:130 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:115 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 peering 2024-09-15T03:54:12.510195+0000 34'67 72:134 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:08:25.470832+0000 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:102 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.290 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:126 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.511899+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 68:107 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 peering 2024-09-15T03:54:12.512060+0000 34'66 72:131 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T12:11:31.753312+0000 4 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.508031+0000 32'65 72:112 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.610285+0000 32'65 72:117 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 68:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:116 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.514737+0000 34'66 72:124 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.291 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.292 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.292 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.292 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:14.292 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:14.292 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:14.299 INFO:tasks.workunit.client.0.smithi138.stdout:wait for peering 2024-09-15T03:54:14.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-09-15T03:54:14.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:14.636 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:14.636 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.512270+0000 34'66 72:122 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.508138+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 68:92 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 68:106 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:127 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.510788+0000 34'67 72:125 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:102 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 68:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:14.637 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 peering 2024-09-15T03:54:12.607015+0000 34'66 72:130 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:115 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 peering 2024-09-15T03:54:12.510195+0000 34'67 72:134 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:08:25.470832+0000 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:102 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:126 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.511899+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 68:107 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 peering 2024-09-15T03:54:12.512060+0000 34'66 72:131 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T12:11:31.753312+0000 4 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.508031+0000 32'65 72:112 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.610285+0000 32'65 72:117 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.638 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 68:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:116 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.514737+0000 34'66 72:124 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:14.639 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:14.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-09-15T03:54:14.759 INFO:tasks.workunit.client.0.smithi138.stdout:killing divergent 1 2024-09-15T03:54:14.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-09-15T03:54:14.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:15.095 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:15.095 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.512270+0000 34'66 72:122 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.095 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050913+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.095 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.508138+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051591+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051276+0000 32'65 68:92 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051646+0000 34'67 68:120 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.049847+0000 34'67 68:106 [2,0] 2 [2,0] 2 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.051032+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048762+0000 34'67 67:127 [0,2] 0 [0,2] 0 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.510788+0000 34'67 72:125 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050180+0000 32'65 67:102 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 5 0 0 2415 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049118+0000 34'66 68:127 [2,0] 2 [2,0] 2 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:29:32.236621+0000 4 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 peering 2024-09-15T03:54:12.607015+0000 34'66 72:130 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:22.187335+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050168+0000 34'66 67:115 [0,2] 0 [0,2] 0 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.096 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 peering 2024-09-15T03:54:12.510195+0000 34'67 72:134 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T08:08:25.470832+0000 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050235+0000 32'65 67:102 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 3 0 0 1449 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049527+0000 34'66 67:126 [0,2] 0 [0,2] 0 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:41:35.135757+0000 2 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 peering 2024-09-15T03:54:12.511899+0000 34'67 72:125 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 0 4 0 0 1618 0 0 20 48 20 active+undersized+degraded 2024-09-15T03:53:10.049104+0000 67'68 68:107 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 peering 2024-09-15T03:54:12.512060+0000 34'66 72:131 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T12:11:31.753312+0000 4 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.050447+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.508031+0000 32'65 72:112 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 peering 2024-09-15T03:54:12.610285+0000 32'65 72:117 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051335+0000 34'67 68:118 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.048895+0000 34'67 67:116 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 peering 2024-09-15T03:54:12.514737+0000 34'66 72:124 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.048758+0000 34'66 67:126 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.050709+0000 32'65 68:104 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 4 0 0 1947 0 0 19 48 19 active+undersized+degraded 2024-09-15T03:53:10.051036+0000 34'67 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 2 0 0 981 0 0 17 48 17 active+undersized+degraded 2024-09-15T03:53:10.052034+0000 32'65 68:106 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.051339+0000 34'66 68:105 [2,0] 2 [2,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 0 3 0 0 1464 0 0 18 48 18 active+undersized+degraded 2024-09-15T03:53:10.049807+0000 34'66 67:115 [0,2] 0 [0,2] 0 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:15.098 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:15.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T03:54:15.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:54:15.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:54:15.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:54:15.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:54:15.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.32635.out 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.32635.out 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:54:15.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:54:15.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:54:15.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T03:54:15.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.32635.out 2024-09-15T03:54:16.581 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.1b info 1.1b( v 34'66 lc 0'0 (31'48,34'66] local-lis/les=72/73 n=3 ec=33/23 lis/c=72/39 les/c/f=73/40/0 sis=72) 2024-09-15T03:54:16.582 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-09-15T03:54:16.618 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-09-15T03:54:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd677bd3:::existing_35:head# 2024-09-15T03:54:16.659 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-09-15T03:54:16.673 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T03:54:16.673 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T03:54:16.673 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T03:54:16.673 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-09-15T03:54:16.840 INFO:tasks.workunit.client.0.smithi138.stderr:Remove successful 2024-09-15T03:54:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.32635.out 2024-09-15T03:54:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:54:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:54:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:54:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:54:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T03:54:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.32635.out 2024-09-15T03:54:18.226 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e74 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-09-15T03:54:18.226 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.1b 2024-09-15T03:54:18.226 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-09-15T03:54:18.226 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:54:18.312 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-09-15T03:54:18.312 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:54:18.350 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd677bd3:::existing_35:head# 2024-09-15T03:54:18.350 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:54:18.379 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-09-15T03:54:18.379 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T03:54:18.412 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 73 info 1.1b( v 34'66 lc 0'0 (31'48,34'66] local-lis/les=72/73 n=3 ec=33/23 lis/c=72/39 les/c/f=73/40/0 sis=72) 2024-09-15T03:54:18.446 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T03:54:18.961 INFO:tasks.workunit.client.0.smithi138.stdout:reviving divergent 1 2024-09-15T03:54:18.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-09-15T03:54:18.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.179569+0000 34'66 73:132 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.182982+0000 34'66 73:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.178472+0000 34'67 73:135 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.184256+0000 34'66 73:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.183179+0000 32'65 73:110 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:37.132479+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 1 1 0 0 1947 0 0 19 48 19 active+recovery_wait+degraded 2024-09-15T03:54:13.180810+0000 34'67 73:119 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.306 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.182490+0000 34'67 73:134 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.179969+0000 32'65 73:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 5 0 0 0 0 2430 0 0 20 48 20 active+clean 2024-09-15T03:54:13.183712+0000 74'68 74:135 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.177759+0000 34'67 73:135 [2,0,1] 2 [2,0,1] 2 34'67 2024-09-15T03:52:37.796053+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.180388+0000 32'65 73:116 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 active+clean 2024-09-15T03:54:13.182335+0000 34'66 73:131 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T12:44:14.586721+0000 4 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 active+clean 2024-09-15T03:54:13.529517+0000 34'66 74:140 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:34.112510+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.179767+0000 34'66 73:121 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 active+clean 2024-09-15T03:54:13.179827+0000 34'67 73:144 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T09:29:01.823801+0000 0 0 2024-09-15T03:54:19.307 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.183255+0000 32'65 73:122 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:30.221117+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:54:13.181318+0000 34'66 73:120 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T07:14:45.136548+0000 2 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.177577+0000 34'67 73:135 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 1 1 0 0 1618 0 0 20 48 20 active+recovery_wait+degraded 2024-09-15T03:54:13.179860+0000 67'68 73:131 [1,2,0] 1 [1,2,0] 1 38'69 2024-09-15T03:52:35.097400+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 active+clean 2024-09-15T03:54:13.177595+0000 34'66 73:141 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T05:11:24.902717+0000 4 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.183884+0000 34'66 73:127 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:36.285595+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.308 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 1 0 0 981 0 0 17 48 17 active+recovering+degraded 2024-09-15T03:54:13.182818+0000 32'65 73:121 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.529520+0000 32'65 74:127 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.180692+0000 34'67 73:122 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.111932+0000 34'67 73:126 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:34.259105+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.111053+0000 34'66 73:134 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:31.803983+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.179286+0000 34'66 73:123 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.182934+0000 32'65 73:120 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:13.180222+0000 34'67 73:127 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:32.279615+0000 34'67 2024-09-15T03:52:32.279615+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:13.183364+0000 32'65 73:106 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:13.184078+0000 34'66 73:111 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.309 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 1 1 0 0 1464 0 0 18 48 18 active+recovery_wait+degraded 2024-09-15T03:54:13.179505+0000 34'66 73:132 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:19.310 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:19.310 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:19.310 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:19.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-09-15T03:54:19.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:54:19.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:54:19.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:54:19.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:54:19.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:54:19.323 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:54:19.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:54:19.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:54:19.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:54:19.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:54:19.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:54:19.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:54:19.326 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:54:19.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:54:19.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:54:19.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:54:19.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:54:19.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:54:19.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:54:19.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:54:19.349 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:19.348+0000 7f099c613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:19.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:19.350+0000 7f099c613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:19.352 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:19.351+0000 7f099c613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:19.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:19.695 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:19.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:19.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:54:19.865 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:19.865+0000 7f099c613440 -1 Falling back to public interface 2024-09-15T03:54:20.090 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 49 last_clean_interval [10,46) [v2:127.0.0.1:6818/71735,v1:127.0.0.1:6819/71735] [v2:127.0.0.1:6820/71735,v1:127.0.0.1:6821/71735] exists,up dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:54:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:54:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:54:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:54:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-09-15T03:54:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:20.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:20.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:20.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:54:20.447 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 72 up_thru 72 down_at 49 last_clean_interval [10,46) [v2:127.0.0.1:6818/71735,v1:127.0.0.1:6819/71735] [v2:127.0.0.1:6820/71735,v1:127.0.0.1:6821/71735] exists,up dd75342f-e3b1-4fab-8134-909af98dedc4 2024-09-15T03:54:20.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:54:20.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:54:20.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:54:20.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-09-15T03:54:22.691 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:22.690+0000 7f099c613440 -1 osd.1 74 log_to_monitors true 2024-09-15T03:54:40.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-09-15T03:54:40.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok dump_ops_in_flight 2024-09-15T03:54:40.532 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:54:40.533 INFO:tasks.workunit.client.0.smithi138.stdout: "ops": [], 2024-09-15T03:54:40.533 INFO:tasks.workunit.client.0.smithi138.stdout: "num_ops": 0 2024-09-15T03:54:40.533 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:54:40.541 INFO:tasks.workunit.client.0.smithi138.stdout:allowing recovery 2024-09-15T03:54:40.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-09-15T03:54:40.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:40.880 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:40.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.691177+0000 34'66 79:148 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697286+0000 34'66 79:135 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.689312+0000 34'67 79:151 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.692525+0000 34'66 79:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696962+0000 32'65 79:122 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:37.132479+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 1 1 0 0 1947 0 0 19 48 19 active+recovery_wait+degraded 2024-09-15T03:54:24.693194+0000 34'67 79:130 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.693123+0000 34'67 79:146 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696369+0000 32'65 79:132 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 5 0 0 0 0 2430 0 0 20 48 20 active+clean 2024-09-15T03:54:24.695562+0000 74'68 79:147 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.689691+0000 34'67 79:151 [2,0,1] 2 [2,0,1] 2 34'67 2024-09-15T03:52:37.796053+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.695207+0000 32'65 79:128 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693323+0000 34'66 79:143 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:03:58.173068+0000 4 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693337+0000 34'66 79:157 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:34.112510+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.696591+0000 34'66 79:133 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.881 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 active+clean 2024-09-15T03:54:24.690656+0000 34'67 79:160 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:18:24.789314+0000 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.691544+0000 32'65 79:134 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:30.221117+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697477+0000 34'66 79:132 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:57:52.190415+0000 2 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.695227+0000 34'67 79:151 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 1 1 0 0 1618 0 0 20 48 20 active+recovery_wait+degraded 2024-09-15T03:54:24.695520+0000 67'68 79:142 [1,2,0] 1 [1,2,0] 1 38'69 2024-09-15T03:52:35.097400+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 active+clean 2024-09-15T03:54:24.689904+0000 34'66 79:157 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:32:06.827251+0000 4 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.695020+0000 34'66 79:139 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:36.285595+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 1 0 0 981 0 0 17 48 17 active+recovering+degraded 2024-09-15T03:54:24.697753+0000 32'65 79:137 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.693537+0000 32'65 79:144 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.694312+0000 34'67 79:134 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.697695+0000 34'67 79:138 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:34.259105+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.691602+0000 34'66 79:150 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:31.803983+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697411+0000 34'66 79:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.882 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.692896+0000 32'65 79:132 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.695412+0000 34'67 79:139 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:32.279615+0000 34'67 2024-09-15T03:52:32.279615+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696684+0000 32'65 79:118 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693510+0000 34'66 79:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 1 1 0 0 1464 0 0 18 48 18 active+recovery_wait+degraded 2024-09-15T03:54:24.697194+0000 34'66 79:143 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:40.883 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:40.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T03:54:40.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T03:54:40.993 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:54:41.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T03:54:41.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T03:54:41.105 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:54:41.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-09-15T03:54:41.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T03:54:41.218 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:54:41.226 INFO:tasks.workunit.client.0.smithi138.stdout:reading divergent objects 2024-09-15T03:54:41.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-09-15T03:54:41.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.1f 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.691177+0000 34'66 79:148 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.1e 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697286+0000 34'66 79:135 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.1d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.689312+0000 34'67 79:151 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.b 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.692525+0000 34'66 79:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.a 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696962+0000 32'65 79:122 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:37.132479+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.9 4 1 1 0 0 1947 0 0 19 48 19 active+recovery_wait+degraded 2024-09-15T03:54:24.693194+0000 34'67 79:130 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.8 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.693123+0000 34'67 79:146 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:26.370929+0000 34'67 2024-09-15T03:52:26.370929+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696369+0000 32'65 79:132 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 5 0 0 0 0 2430 0 0 20 48 20 active+clean 2024-09-15T03:54:24.695562+0000 74'68 79:147 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:27.234166+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.568 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.689691+0000 34'67 79:151 [2,0,1] 2 [2,0,1] 2 34'67 2024-09-15T03:52:37.796053+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.695207+0000 32'65 79:128 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:24.214761+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 5 0 0 0 0 2415 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693323+0000 34'66 79:143 [1,2,0] 1 [1,2,0] 1 32'65 2024-09-15T03:52:21.189096+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T14:03:58.173068+0000 4 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 3 0 0 0 0 1465 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693337+0000 34'66 79:157 [0,1,2] 0 [0,1,2] 0 34'66 2024-09-15T03:52:34.112510+0000 34'66 2024-09-15T03:52:22.187335+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.c 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.696591+0000 34'66 79:133 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:28.319741+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 2 0 0 0 0 966 0 0 19 48 19 active+clean 2024-09-15T03:54:24.690656+0000 34'67 79:160 [2,0,1] 2 [2,0,1] 2 32'65 2024-09-15T03:52:20.215905+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T04:18:24.789314+0000 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.1c 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.691544+0000 32'65 79:134 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:30.221117+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 3 0 0 0 0 1449 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697477+0000 34'66 79:132 [1,0,2] 1 [1,0,2] 1 32'65 2024-09-15T03:52:19.286087+0000 32'65 2024-09-15T03:52:19.286087+0000 0 1 periodic scrub scheduled @ 2024-09-16T10:57:52.190415+0000 2 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.d 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.695227+0000 34'67 79:151 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.e 4 1 1 0 0 1618 0 0 20 48 20 active+recovery_wait+degraded 2024-09-15T03:54:24.695520+0000 67'68 79:142 [1,2,0] 1 [1,2,0] 1 38'69 2024-09-15T03:52:35.097400+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.f 4 0 0 0 0 1932 0 0 18 48 18 active+clean 2024-09-15T03:54:24.689904+0000 34'66 79:157 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:27.792541+0000 0'0 2024-09-15T03:51:55.456175+0000 0 1 periodic scrub scheduled @ 2024-09-16T06:32:06.827251+0000 4 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.10 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.695020+0000 34'66 79:139 [1,0,2] 1 [1,0,2] 1 34'66 2024-09-15T03:52:36.285595+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.569 INFO:tasks.workunit.client.0.smithi138.stdout:1.11 2 0 1 0 0 981 0 0 17 48 17 active+recovering+degraded 2024-09-15T03:54:24.697753+0000 32'65 79:137 [2,0,1] 2 [2,0,1] 2 38'66 2024-09-15T03:52:30.778236+0000 38'66 2024-09-15T03:52:30.778236+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.12 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.693537+0000 32'65 79:144 [0,1,2] 0 [0,1,2] 0 32'65 2024-09-15T03:52:29.101357+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.13 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.694312+0000 34'67 79:134 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.14 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.697695+0000 34'67 79:138 [1,0,2] 1 [1,0,2] 1 34'67 2024-09-15T03:52:34.259105+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.15 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.691602+0000 34'66 79:150 [2,0,1] 2 [2,0,1] 2 34'66 2024-09-15T03:52:31.803983+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.16 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.697411+0000 34'66 79:135 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.17 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.692896+0000 32'65 79:132 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.18 4 0 0 0 0 1947 0 0 19 48 19 active+clean 2024-09-15T03:54:24.695412+0000 34'67 79:139 [1,2,0] 1 [1,2,0] 1 34'67 2024-09-15T03:52:32.279615+0000 34'67 2024-09-15T03:52:32.279615+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.19 2 0 0 0 0 981 0 0 17 48 17 active+clean 2024-09-15T03:54:24.696684+0000 32'65 79:118 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.1a 3 0 0 0 0 1464 0 0 18 48 18 active+clean 2024-09-15T03:54:24.693510+0000 34'66 79:123 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:1.1b 3 1 1 0 0 1464 0 0 18 48 18 active+recovery_wait+degraded 2024-09-15T03:54:24.697194+0000 34'66 79:143 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:51:55.456175+0000 0'0 2024-09-15T03:51:55.456175+0000 0 0 queued for scrub 0 0 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:54:41.570 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:54:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:54:41.579 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-09-15T03:54:41.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-09-15T03:54:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:54:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T03:54:41.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:54:41.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T03:54:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:54:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T03:54:41.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:54:41.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T03:54:41.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-09-15T03:54:41.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:54:41.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-09-15T03:54:41.731 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test/newobject_1: (2) No such file or directory 2024-09-15T03:54:41.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-09-15T03:54:41.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-09-15T03:54:41.762 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test/newobject_2: (2) No such file or directory 2024-09-15T03:54:41.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-09-15T03:54:41.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T03:54:41.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.612+0000 7fedde3b6640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 38'69 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-09-15T03:52:29.259921+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.612+0000 7fedde3b6640 20 _merge_object_divergent_entries: keeping 38'69 (31'4) delete 1:70b3bf12:::existing_4:head by client.4636.0:1 2024-09-15T03:52:29.259921+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.612+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.612+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 31'4 first_divergent_update: 38'69 last_divergent_update: 38'69 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.612+0000 7fedde3b6640 10 _merge_object_divergent_entries: more recent entry found: 67'68 (31'4) modify 1:70b3bf12:::existing_4:head by client.4754.0:1 2024-09-15T03:53:30.356301+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 38'68 (0'0) modify 1:75ac5091:::newobject_2:head by client.4626.0:1 2024-09-15T03:52:29.247044+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 20 _merge_object_divergent_entries: keeping 38'68 (0'0) modify 1:75ac5091:::newobject_2:head by client.4626.0:1 2024-09-15T03:52:29.247044+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:54:41.785 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 38'68 last_divergent_update: 38'68 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.613+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 38'68 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-15T03:52:29.254941+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 20 _merge_object_divergent_entries: keeping 38'68 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-15T03:52:29.254941+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 31'1 first_divergent_update: 38'68 last_divergent_update: 38'68 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 38'68 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 38'68 (31'1) modify 1:d858c035:::existing_1:head by client.4631.0:1 2024-09-15T03:52:29.254941+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 31'1 <= info.log_tail 31'48 2024-09-15T03:54:41.786 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 38'67 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4630.0:1 2024-09-15T03:52:29.254252+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 20 _merge_object_divergent_entries: keeping 38'67 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4630.0:1 2024-09-15T03:52:29.254252+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 38'67 last_divergent_update: 38'67 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.705+0000 7fedde3b6640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 38'68 (31'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-09-15T03:52:29.237367+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 20 _merge_object_divergent_entries: keeping 38'68 (31'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-09-15T03:52:29.237367+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 31'2 first_divergent_update: 38'68 last_divergent_update: 38'68 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:54:41.787 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 38'68 2024-09-15T03:54:41.795 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 38'68 (31'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-09-15T03:52:29.237367+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:54:41.795 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-09-15T03:54:41.795 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.706+0000 7fedde3b6640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 31'2 <= info.log_tail 31'48 2024-09-15T03:54:41.795 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.795 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 20 _merge_object_divergent_entries: keeping 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 38'66 last_divergent_update: 38'66 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 38'66 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:54:12.957+0000 7feddf3b8640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'48 2024-09-15T03:54:41.796 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 20 _merge_object_divergent_entries: keeping 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 31'3 first_divergent_update: 38'66 last_divergent_update: 38'66 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 38'66 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 38'66 (31'3) delete 1:8881e6d8:::existing_3:head by client.4637.0:1 2024-09-15T03:52:29.262028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2024-09-15T03:54:41.797 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-09-15T03:54:41.798 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.2.log:2024-09-15T03:54:12.706+0000 7f4c2d7c5640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 31'3 <= info.log_tail 31'48 2024-09-15T03:54:41.798 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T03:54:41.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:54:41.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-09-15T03:54:41.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:54:41.834 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T03:54:41.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-09-15T03:54:41.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.32635.out 2024-09-15T03:54:41.835 INFO:tasks.workunit.client.0.smithi138.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.32635.out': No such file or directory 2024-09-15T03:54:41.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-09-15T03:54:41.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T03:54:41.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T03:54:42.396 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T03:54:42.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-09-15T03:54:42.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:54:42.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:54:42.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:54:42.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:54:42.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:54:47.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:54:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:54:47.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:54:47.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:54:47.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:54:47.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:54:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:54:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:54:47.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:54:47.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:54:47.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:54:47.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:54:47.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:54:47.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:54:47.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:54:47.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:54:47.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:54:47.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:47.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:47.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-09-15T03:54:47.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:54:47.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:54:47.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:54:47.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:54:47.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:54:47.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:54:47.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:54:47.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:54:47.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:54:47.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:54:47.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:54:47.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:54:47.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:54:47.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:54:47.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:54:47.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:54:47.784 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:54:47.784 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:47.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:47.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:54:47.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:54:47.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:54:47.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-09-15T03:54:47.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:54:47.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:47.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:47.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.32635 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-09-15T03:54:47.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-09-15T03:54:47.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-09-15T03:54:47.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-09-15T03:54:47.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:54:47.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:54:47.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:54:47.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-09-15T03:54:47.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:48.157 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:48.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:54:48.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:54:48.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:54:48.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:54:48.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:54:48.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:54:48.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:54:48.416 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:54:48.416 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:54:48.416 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:54:48.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get fsid 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:54:48.499 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-mon.a.asok 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:54:48.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.32635/ceph-mon.a.asok config get mon_host 2024-09-15T03:54:48.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-09-15T03:54:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:49.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:49.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:49.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:54:49.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:54:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T03:54:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-09-15T03:54:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:54:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:54:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:54:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:54:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:54:49.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:54:49.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:54:49.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 14218549-25fd-49a0-801f-6454193e2a35' 2024-09-15T03:54:49.464 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:54:49.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:54:49.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJWuZmA+OWHBAAIr37wkS8ii0Kjc87aB1W6A== 2024-09-15T03:54:49.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJWuZmA+OWHBAAIr37wkS8ii0Kjc87aB1W6A=="}' 2024-09-15T03:54:49.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14218549-25fd-49a0-801f-6454193e2a35 -i td/divergent-priors/0/new.json 2024-09-15T03:54:49.771 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:49.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-09-15T03:54:49.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJWuZmA+OWHBAAIr37wkS8ii0Kjc87aB1W6A== --osd-uuid 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:54:49.813 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.811+0000 7f925a933440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:49.816 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.815+0000 7f925a933440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:49.817 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.816+0000 7f925a933440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:49.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.816+0000 7f925a933440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:54:49.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.816+0000 7f925a933440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:54:49.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.816+0000 7f925a933440 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-09-15T03:54:49.857 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:49.855+0000 7f925a933440 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-09-15T03:54:54.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-09-15T03:54:54.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:54:54.950 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:54:54.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:54:54.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:54:55.470 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:54:55.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:54:55.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:54:55.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:54:55.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:54:55.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:54:55.500 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:55.499+0000 7fc55300b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:55.502 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:55.501+0000 7fc55300b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:55.503 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:55.502+0000 7fc55300b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:54:55.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:54:55.826 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:54:55.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:55.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:54:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:56.020+0000 7fc55300b440 -1 Falling back to public interface 2024-09-15T03:54:56.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:54:57.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:54:57.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:57.183 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:54:57.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:54:57.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:57.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:54:57.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:54:58.479 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:54:58.478+0000 7fc55300b440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:54:58.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:54:58.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:54:58.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:54:58.575 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:54:58.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:54:58.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:54:59.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:00.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:00.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:00.077 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:55:00.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:55:00.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:00.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:01.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:01.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:01.478 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:55:01.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:55:01.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:01.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:01.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:02.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:02.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:02.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:55:02.868 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:55:02.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:02.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:03.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:04.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:04.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:04.334 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:55:04.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:55:04.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:04.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:04.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:05.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:05.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:05.695 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:55:05.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:55:05.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:05.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:06.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:07.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:07.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:07.189 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:55:07.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:55:07.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:07.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:08.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:08.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:08.554 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:55:08.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:55:08.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:08.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:08.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:09.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:09.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:09.929 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:55:09.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:55:09.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:09.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:10.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:11.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:11.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:11.741 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:55:11.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:55:11.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:11.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:13.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:13.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:13.554 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:55:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:55:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:14.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:15.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:15.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:15.392 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:55:15.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:55:15.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:15.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:16.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:17.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:17.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:17.450 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:55:17.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:55:17.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:17.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:18.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:19.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:19.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:19.145 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:55:19.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:55:19.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:19.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:20.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:21.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:21.104 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:55:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:55:21.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:21.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:22.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:22.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:22.739 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:55:22.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:55:22.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:22.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:55:23.105 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/75157,v1:127.0.0.1:6803/75157] [v2:127.0.0.1:6804/75157,v1:127.0.0.1:6805/75157] exists,up 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:55:23.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:55:23.110 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:55:23.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:55:23.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:55:23.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:55:23.112 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:55:23.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ee84699d-c2a4-4409-8483-0497fc15a0ca' 2024-09-15T03:55:23.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:55:23.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrWuZmnOikBxAAaXyOdF28880OhSlWp9/zOQ== 2024-09-15T03:55:23.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrWuZmnOikBxAAaXyOdF28880OhSlWp9/zOQ=="}' 2024-09-15T03:55:23.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ee84699d-c2a4-4409-8483-0497fc15a0ca -i td/divergent-priors/1/new.json 2024-09-15T03:55:23.607 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:55:23.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-09-15T03:55:23.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrWuZmnOikBxAAaXyOdF28880OhSlWp9/zOQ== --osd-uuid ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:55:23.650 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.649+0000 7fc09ac71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:23.652 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.651+0000 7fc09ac71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:23.653 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.652+0000 7fc09ac71440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:23.654 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.653+0000 7fc09ac71440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:55:23.654 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.653+0000 7fc09ac71440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:55:23.654 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.653+0000 7fc09ac71440 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-09-15T03:55:23.731 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:23.731+0000 7fc09ac71440 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-09-15T03:55:29.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-09-15T03:55:29.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:55:29.384 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T03:55:29.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T03:55:29.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:55:29.878 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:55:29.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T03:55:29.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:55:29.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:55:29.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:55:29.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:55:29.909 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:29.908+0000 7f8097a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:29.935 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:29.934+0000 7f8097a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:29.936 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:29.935+0000 7f8097a08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:30.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T03:55:30.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:55:30.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:55:30.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:30.465 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:30.465+0000 7f8097a08440 -1 Falling back to public interface 2024-09-15T03:55:30.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:31.672 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:55:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:55:31.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:31.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:32.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:32.925 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:32.924+0000 7f8097a08440 -1 osd.1 0 log_to_monitors true 2024-09-15T03:55:33.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:33.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:33.079 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:55:33.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:55:33.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:33.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:33.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:34.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:34.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:34.626 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:55:34.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:55:34.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:34.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:35.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:36.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:36.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:36.042 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:55:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:55:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:36.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:37.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:37.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:37.413 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:55:37.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:55:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:38.872 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:55:38.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:55:38.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:38.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:39.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:40.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:40.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:40.289 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:55:40.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:55:40.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:40.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:40.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:41.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:41.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:41.824 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:55:41.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:55:41.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:41.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:42.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:43.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:43.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:43.192 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:55:43.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:55:43.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:43.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:43.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:44.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:44.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:44.663 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:55:44.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:55:44.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:44.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:45.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:46.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:46.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:46.467 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:55:46.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:55:46.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:46.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:48.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:48.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:48.161 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:55:48.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:55:48.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:48.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:49.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:50.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:50.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:50.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:55:50.260 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:55:50.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:50.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:50.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:51.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:51.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:51.995 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:55:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:55:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:52.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:53.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:53.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:53.782 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:55:53.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:55:53.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:53.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:54.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:55:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:55:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:55:55.552 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:55:55.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:55:55.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:55:55.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:55:56.037 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/76922,v1:127.0.0.1:6811/76922] [v2:127.0.0.1:6812/76922,v1:127.0.0.1:6813/76922] exists,up ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:55:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:55:56.043 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:55:56.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4' 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:55:56.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:55:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMWuZmGAmqAxAAN3F5XJB4FQfcsikldc3xIw== 2024-09-15T03:55:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMWuZmGAmqAxAAN3F5XJB4FQfcsikldc3xIw=="}' 2024-09-15T03:55:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 -i td/divergent-priors/2/new.json 2024-09-15T03:55:56.506 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:55:56.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-09-15T03:55:56.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMWuZmGAmqAxAAN3F5XJB4FQfcsikldc3xIw== --osd-uuid f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.549+0000 7fe9ee61c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:56.551 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.551+0000 7fe9ee61c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:56.552 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.552+0000 7fe9ee61c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:55:56.553 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.552+0000 7fe9ee61c440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:55:56.553 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.552+0000 7fe9ee61c440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:55:56.553 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.552+0000 7fe9ee61c440 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-09-15T03:55:56.646 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:55:56.646+0000 7fe9ee61c440 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-09-15T03:56:02.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-09-15T03:56:02.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:56:02.810 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T03:56:02.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T03:56:02.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:56:03.306 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:56:03.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T03:56:03.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:56:03.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:56:03.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:56:03.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:56:03.336 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:56:03.336+0000 7fefae60e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:56:03.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:56:03.359+0000 7fefae60e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:56:03.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:56:03.360+0000 7fefae60e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:56:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:03.733 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:56:03.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:56:03.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:03.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:03.893 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:56:03.893+0000 7fefae60e440 -1 Falling back to public interface 2024-09-15T03:56:04.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:05.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:05.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:05.168 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:56:05.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:56:05.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:05.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:05.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:06.366 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:56:06.365+0000 7fefae60e440 -1 osd.2 0 log_to_monitors true 2024-09-15T03:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:06.525 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:07.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:08.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:08.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:08.019 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:56:08.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:56:08.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:08.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:08.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:09.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:09.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:09.425 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:56:09.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:56:09.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:09.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:09.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:10.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:10.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:10.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T03:56:10.888 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T03:56:10.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:10.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:11.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:12.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:12.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:12.307 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T03:56:12.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T03:56:12.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:12.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:12.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:13.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:13.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:13.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T03:56:13.689 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T03:56:13.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:13.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:15.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:15.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:15.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T03:56:15.156 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T03:56:15.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:15.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:16.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:16.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:16.578 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T03:56:16.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T03:56:16.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:16.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:16.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:17.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:17.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:17.943 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T03:56:17.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T03:56:17.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:17.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:18.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:19.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:19.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:19.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T03:56:19.359 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T03:56:19.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:19.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:21.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:21.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:21.297 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T03:56:21.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T03:56:21.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:21.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:22.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:23.182 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T03:56:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T03:56:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:23.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:24.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:24.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:24.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T03:56:24.913 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T03:56:24.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:24.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:25.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:26.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:26.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:26.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T03:56:26.413 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T03:56:26.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:26.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:27.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:28.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:28.258 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T03:56:28.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T03:56:28.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:28.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:29.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:56:30.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:56:30.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:56:30.208 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T03:56:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T03:56:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:56:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:56:30.572 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/78651,v1:127.0.0.1:6819/78651] [v2:127.0.0.1:6820/78651,v1:127.0.0.1:6821/78651] exists,up f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:56:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:56:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:56:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:56:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-09-15T03:56:31.288 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T03:56:31.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-09-15T03:56:32.001 INFO:tasks.workunit.client.0.smithi138.stderr:noin is set 2024-09-15T03:56:32.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-09-15T03:56:32.821 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T03:56:32.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-09-15T03:56:32.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: create_ec_pool: local pool_name=test 2024-09-15T03:56:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2407: create_ec_pool: shift 2024-09-15T03:56:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2408: create_ec_pool: local allow_overwrites=true 2024-09-15T03:56:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2409: create_ec_pool: shift 2024-09-15T03:56:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-09-15T03:56:33.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-09-15T03:56:33.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-15T03:56:34.041 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T03:56:34.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T03:56:35.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: create_ec_pool: '[' true = true ']' 2024-09-15T03:56:35.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-09-15T03:56:35.840 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 allow_ec_overwrites to true 2024-09-15T03:56:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: create_ec_pool: wait_for_clean 2024-09-15T03:56:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:56:35.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:56:35.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:56:35.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:56:35.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:56:35.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:56:35.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:56:35.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:56:35.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:56:35.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:56:35.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:56:35.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:56:35.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:56:35.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:56:35.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:56:36.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:56:36.299 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:56:36.299 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:56:36.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:56:36.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:36.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:56:36.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836502 2024-09-15T03:56:36.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836502 2024-09-15T03:56:36.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502' 2024-09-15T03:56:36.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:36.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:56:36.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-15T03:56:36.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-15T03:56:36.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672972' 2024-09-15T03:56:36.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:56:36.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-15T03:56:36.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-15T03:56:36.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836502 1-42949672972 2-64424509452' 2024-09-15T03:56:36.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:36.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836502 2024-09-15T03:56:36.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:36.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:56:36.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836502 2024-09-15T03:56:36.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:36.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836502 2024-09-15T03:56:36.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-09-15T03:56:36.660 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836502 2024-09-15T03:56:36.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:37.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836502 2024-09-15T03:56:37.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:56:38.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:56:38.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:38.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836502 2024-09-15T03:56:38.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:38.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-15T03:56:38.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:38.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:56:38.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-15T03:56:38.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:38.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-15T03:56:38.380 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672972 2024-09-15T03:56:38.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-15T03:56:38.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:56:38.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-15T03:56:38.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:38.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-15T03:56:38.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:38.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:56:38.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-15T03:56:38.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:38.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-15T03:56:38.741 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509452 2024-09-15T03:56:38.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T03:56:38.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:56:39.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-15T03:56:39.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:56:39.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:56:39.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:56:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:56:40.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:56:40.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:56:40.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:56:40.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:56:40.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:56:40.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:56:40.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:56:40.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: create_ec_pool: return 0 2024-09-15T03:56:40.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-09-15T03:56:40.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:56:40.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:56:40.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:56:40.870 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:56:40.871 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:56:40.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:56:40.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:40.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:56:40.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-15T03:56:40.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-15T03:56:40.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T03:56:40.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:40.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:56:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672974 2024-09-15T03:56:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672974 2024-09-15T03:56:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672974' 2024-09-15T03:56:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:41.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:56:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509455 2024-09-15T03:56:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509455 2024-09-15T03:56:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672974 2-64424509455' 2024-09-15T03:56:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:41.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-15T03:56:41.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:41.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:56:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-15T03:56:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:41.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-15T03:56:41.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T03:56:41.213 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836505 2024-09-15T03:56:41.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:41.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-15T03:56:41.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:56:42.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:56:42.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:42.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836505 2024-09-15T03:56:42.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:56:43.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T03:56:43.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:44.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836505 2024-09-15T03:56:44.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:44.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672974 2024-09-15T03:56:44.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:44.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:56:44.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672974 2024-09-15T03:56:44.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:44.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672974 2024-09-15T03:56:44.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-09-15T03:56:44.359 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672974 2024-09-15T03:56:44.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:56:44.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672974 2024-09-15T03:56:44.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:44.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509455 2024-09-15T03:56:44.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:44.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:56:44.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509455 2024-09-15T03:56:44.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:44.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509455 2024-09-15T03:56:44.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-15T03:56:44.727 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509455 2024-09-15T03:56:44.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509455 -lt 64424509455 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:56:45.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:56:45.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T03:56:45.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:56:45.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:56:45.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:56:45.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:56:45.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:45.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:56:45.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-15T03:56:45.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-15T03:56:45.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-15T03:56:45.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:45.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:56:45.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672977 2024-09-15T03:56:45.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672977 2024-09-15T03:56:45.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672977' 2024-09-15T03:56:45.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:56:45.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:56:45.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509458 2024-09-15T03:56:45.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509458 2024-09-15T03:56:45.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672977 2-64424509458' 2024-09-15T03:56:45.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:45.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-15T03:56:45.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:45.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:56:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-15T03:56:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:45.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-15T03:56:45.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-15T03:56:45.969 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836508 2024-09-15T03:56:45.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:46.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-15T03:56:46.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:56:47.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:56:47.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:56:47.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836508 -lt 21474836508 2024-09-15T03:56:47.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:47.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672977 2024-09-15T03:56:47.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:47.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:56:47.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672977 2024-09-15T03:56:47.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:47.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672977 2024-09-15T03:56:47.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-09-15T03:56:47.727 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672977 2024-09-15T03:56:47.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:56:48.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672977 2024-09-15T03:56:48.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:56:48.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509458 2024-09-15T03:56:48.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:56:48.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:56:48.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509458 2024-09-15T03:56:48.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:56:48.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509458 2024-09-15T03:56:48.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-15T03:56:48.086 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509458 2024-09-15T03:56:48.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:56:48.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-15T03:56:48.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:56:48.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:56:48.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:56:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:56:49.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:56:49.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:56:49.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:56:49.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:56:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:56:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:56:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:56:49.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-09-15T03:56:49.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-09-15T03:56:50.144 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:56:50.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-09-15T03:56:50.155 INFO:tasks.workunit.client.0.smithi138.stdout:primary and soon to be divergent is 1 2024-09-15T03:56:50.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-09-15T03:56:50.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:56:50.491 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:56:50.491 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T03:56:35.244303+0000 0'0 27:11 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:19:56.463237+0000 0 0 2024-09-15T03:56:50.491 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:56:50.491 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:56:50.492 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:56:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-09-15T03:56:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T03:56:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-09-15T03:56:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-09-15T03:56:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-09-15T03:56:50.503 INFO:tasks.workunit.client.0.smithi138.stdout:writing initial objects 2024-09-15T03:56:50.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-09-15T03:56:50.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:50.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-09-15T03:56:50.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:50.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-09-15T03:56:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:50.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-09-15T03:56:51.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:51.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-09-15T03:56:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-09-15T03:56:51.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:51.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-09-15T03:56:51.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:51.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-09-15T03:56:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-09-15T03:56:52.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:52.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-09-15T03:56:52.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:52.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-09-15T03:56:52.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:52.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-09-15T03:56:52.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:52.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-09-15T03:56:53.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-09-15T03:56:53.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:53.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-09-15T03:56:53.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:53.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-09-15T03:56:53.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:53.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-09-15T03:56:53.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:53.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-09-15T03:56:54.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:54.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-09-15T03:56:54.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:54.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-09-15T03:56:54.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:54.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-09-15T03:56:54.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:54.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-09-15T03:56:54.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:54.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-09-15T03:56:55.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:55.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-09-15T03:56:55.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:55.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-09-15T03:56:55.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:55.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-09-15T03:56:55.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:55.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-09-15T03:56:56.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:56.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-09-15T03:56:56.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:56.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-09-15T03:56:56.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:56.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-09-15T03:56:56.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:56.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-09-15T03:56:56.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:56.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-09-15T03:56:57.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:57.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-09-15T03:56:57.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:57.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-09-15T03:56:57.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:57.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-09-15T03:56:57.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:57.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-09-15T03:56:57.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:57.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-09-15T03:56:58.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:58.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-09-15T03:56:58.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:58.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-09-15T03:56:58.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:58.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-09-15T03:56:58.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:58.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-09-15T03:56:59.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:59.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-09-15T03:56:59.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:59.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-09-15T03:56:59.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:59.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-09-15T03:56:59.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:59.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-09-15T03:56:59.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:56:59.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-09-15T03:57:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-09-15T03:57:00.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:00.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-09-15T03:57:00.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:00.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-09-15T03:57:00.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:00.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-09-15T03:57:00.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:00.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-09-15T03:57:01.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:01.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-09-15T03:57:01.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:01.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-09-15T03:57:01.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:01.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-09-15T03:57:01.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:01.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-09-15T03:57:02.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:02.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-09-15T03:57:02.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:02.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-09-15T03:57:02.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:02.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-09-15T03:57:02.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:02.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-09-15T03:57:02.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:02.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-09-15T03:57:03.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:03.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-09-15T03:57:03.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:03.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-09-15T03:57:03.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:03.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-09-15T03:57:03.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:03.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-09-15T03:57:03.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:03.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-09-15T03:57:04.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:04.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-09-15T03:57:04.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:04.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-09-15T03:57:04.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:04.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-09-15T03:57:04.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:04.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-09-15T03:57:05.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:05.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-09-15T03:57:05.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:05.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-09-15T03:57:05.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:05.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-09-15T03:57:05.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:05.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-09-15T03:57:05.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:05.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-09-15T03:57:06.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:06.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-09-15T03:57:06.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:06.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-09-15T03:57:06.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:06.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-09-15T03:57:06.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:06.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-09-15T03:57:06.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:06.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-09-15T03:57:07.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:07.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-09-15T03:57:07.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:07.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-09-15T03:57:07.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:07.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-09-15T03:57:07.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:07.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-09-15T03:57:08.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:08.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-09-15T03:57:08.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:08.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-09-15T03:57:08.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:08.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-09-15T03:57:08.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:08.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-09-15T03:57:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-09-15T03:57:09.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:09.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-09-15T03:57:09.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:09.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-09-15T03:57:09.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:09.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-09-15T03:57:09.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:09.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-09-15T03:57:09.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:09.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-09-15T03:57:10.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:10.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-09-15T03:57:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-09-15T03:57:10.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:10.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-09-15T03:57:10.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:10.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-09-15T03:57:10.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:10.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-09-15T03:57:11.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:11.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-09-15T03:57:11.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:11.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-15T03:57:11.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-09-15T03:57:11.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-15T03:57:11.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:57:11.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-09-15T03:57:11.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:57:11.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:57:11.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:57:11.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:57:11.913 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:57:11.913 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:57:11.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:57:11.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:57:11.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:57:11.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:57:11.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:57:11.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:57:11.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:57:11.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:57:11.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:57:12.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:57:12.356 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:57:12.356 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:57:12.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:57:12.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:12.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:57:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836515 2024-09-15T03:57:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836515 2024-09-15T03:57:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515' 2024-09-15T03:57:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:12.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:57:12.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672985 2024-09-15T03:57:12.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672985 2024-09-15T03:57:12.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515 1-42949672985' 2024-09-15T03:57:12.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:12.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:57:12.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509465 2024-09-15T03:57:12.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509465 2024-09-15T03:57:12.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836515 1-42949672985 2-64424509465' 2024-09-15T03:57:12.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:57:12.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836515 2024-09-15T03:57:12.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:57:12.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:57:12.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836515 2024-09-15T03:57:12.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:57:12.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836515 2024-09-15T03:57:12.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-15T03:57:12.699 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836515 2024-09-15T03:57:12.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:57:13.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836515 2024-09-15T03:57:13.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:57:14.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:57:14.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:57:14.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836515 2024-09-15T03:57:14.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:57:14.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672985 2024-09-15T03:57:14.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:57:14.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T03:57:14.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672985 2024-09-15T03:57:14.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:57:14.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672985 2024-09-15T03:57:14.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-15T03:57:14.473 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672985 2024-09-15T03:57:14.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T03:57:14.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-15T03:57:14.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:57:14.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509465 2024-09-15T03:57:14.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:57:14.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:57:14.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509465 2024-09-15T03:57:14.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:57:14.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509465 2024-09-15T03:57:14.833 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509465 2024-09-15T03:57:14.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-15T03:57:14.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:57:15.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-15T03:57:15.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:57:15.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:15.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:15.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:16.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T03:57:16.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:16.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:16.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:16.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-09-15T03:57:16.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T03:57:16.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-09-15T03:57:16.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-09-15T03:57:16.925 INFO:tasks.workunit.client.0.smithi138.stdout:blackholing osds 0 2 2024-09-15T03:57:16.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:57:17.261 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:57:17.261 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 0 0 0 48300 0 0 16 84 16 active+clean 2024-09-15T03:56:35.244303+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:19:56.463237+0000 0 0 2024-09-15T03:57:17.261 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:57:17.261 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:57:17.262 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:57:17.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:17.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T03:57:17.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok config set objectstore_blackhole 1 2024-09-15T03:57:17.344 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:57:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:57:17.344 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:57:17.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T03:57:17.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok config set objectstore_blackhole 1 2024-09-15T03:57:17.424 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:57:17.425 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-09-15T03:57:17.425 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:57:17.431 INFO:tasks.workunit.client.0.smithi138.stdout:writing divergent object 2024-09-15T03:57:17.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-09-15T03:57:17.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-09-15T03:57:17.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-09-15T03:57:18.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-09-15T03:57:18.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-09-15T03:57:18.616 INFO:tasks.workunit.client.0.smithi138.stdout:created pool test snap snap1 2024-09-15T03:57:18.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-09-15T03:57:18.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-09-15T03:57:18.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-09-15T03:57:28.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-09-15T03:57:28.624 INFO:tasks.workunit.client.0.smithi138.stderr:rados: no process found 2024-09-15T03:57:28.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-09-15T03:57:28.624 INFO:tasks.workunit.client.0.smithi138.stdout:killing all the osds 2024-09-15T03:57:28.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:57:28.973 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:57:28.974 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 active+clean 2024-09-15T03:56:35.244303+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:19:56.463237+0000 0 0 2024-09-15T03:57:28.974 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:57:28.974 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:57:28.974 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:57:28.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-09-15T03:57:28.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:57:28.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:57:28.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:57:28.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:57:28.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:57:29.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:57:29.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T03:57:29.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-09-15T03:57:29.575 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T03:57:29.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T03:57:29.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-09-15T03:57:30.077 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T03:57:30.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-09-15T03:57:30.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-09-15T03:57:30.580 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T03:57:30.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:30.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-09-15T03:57:31.091 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T03:57:31.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:31.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-09-15T03:57:31.591 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T03:57:31.602 INFO:tasks.workunit.client.0.smithi138.stdout:bringing up non_divergent 0 2 2024-09-15T03:57:31.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-09-15T03:57:31.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:57:31.949 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:57:31.949 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 0 0 0 48783 0 0 14 90 14 stale+active+clean 2024-09-15T03:56:35.244303+0000 29'104 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:19:56.463237+0000 0 0 2024-09-15T03:57:31.949 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:57:31.949 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:57:31.949 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:57:31.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:57:31.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:57:31.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:57:31.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:57:31.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:57:31.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:57:31.965 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:57:31.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:57:31.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:57:31.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T03:57:31.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:57:31.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:57:31.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:57:31.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:57:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:31.991+0000 7ff36e24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:31.993 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:31.993+0000 7ff36e24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:31.994 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:31.994+0000 7ff36e24f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:57:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:57:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:57:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:57:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:57:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:57:32.320 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:57:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:32.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:57:32.510 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:32.510+0000 7ff36e24f440 -1 Falling back to public interface 2024-09-15T03:57:32.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:33.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:33.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:33.713 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:57:33.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:57:33.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:33.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:57:34.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:35.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:35.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:35.070 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:57:35.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:57:35.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:35.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:57:35.319 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:35.318+0000 7ff36e24f440 -1 osd.0 28 log_to_monitors true 2024-09-15T03:57:35.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:36.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:36.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:36.429 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:57:36.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:57:36.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:36.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:57:36.786 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/83287,v1:127.0.0.1:6803/83287] [v2:127.0.0.1:6804/83287,v1:127.0.0.1:6805/83287] exists,up 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:57:36.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:57:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:57:36.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:57:36.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:57:36.794 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:57:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:57:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T03:57:36.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:57:36.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:57:36.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:57:36.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:57:36.820 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:36.819+0000 7fd143a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:36.832 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:36.832+0000 7fd143a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:36.833 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:36.833+0000 7fd143a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:57:37.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:37.185 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:57:37.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:57:37.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:37.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:57:37.349 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:37.349+0000 7fd143a09440 -1 Falling back to public interface 2024-09-15T03:57:37.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:38.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:38.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:38.572 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:57:38.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:57:38.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:38.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:57:38.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:39.956 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:57:39.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:57:39.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:39.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:57:40.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:57:40.416 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:57:40.415+0000 7fd143a09440 -1 osd.2 28 log_to_monitors true 2024-09-15T03:57:41.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:57:41.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:57:41.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:57:41.314 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:57:41.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:57:41.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:57:41.673 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/83865,v1:127.0.0.1:6811/83865] [v2:127.0.0.1:6812/83865,v1:127.0.0.1:6813/83865] exists,up f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:57:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:57:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:57:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:57:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-09-15T03:57:42.247 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T03:57:42.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:57:42.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-09-15T03:57:43.017 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already in. 2024-09-15T03:57:43.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-09-15T03:57:48.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-09-15T03:57:48.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-09-15T03:57:48.031 INFO:tasks.workunit.client.0.smithi138.stdout:writing non-divergent object existing_4 2024-09-15T03:57:48.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-09-15T03:57:48.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:57:48.377 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:57:48.377 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 100 0 100 0 0 48300 0 0 16 84 16 active+undersized+degraded 2024-09-15T03:57:43.201348+0000 28'100 42:126 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:37:39.039674+0000 0 0 2024-09-15T03:57:48.377 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:57:48.377 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:57:48.377 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:57:48.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:57:48.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:57:48.491 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:57:48.491 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:57:48.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:57:48.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:57:48.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:57:48.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:48.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:57:49.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=154618822661 2024-09-15T03:57:49.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 154618822661 2024-09-15T03:57:49.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822661' 2024-09-15T03:57:49.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:49.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:57:49.138 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:57:49.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:57:49.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:57:49.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:57:49.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:57:49.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=163208757252 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 163208757252 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-154618822661 2-163208757252' 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-154618822661 2024-09-15T03:57:49.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:57:49.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:57:49.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-154618822661 2024-09-15T03:57:49.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:57:49.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=154618822661 2024-09-15T03:57:49.256 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 154618822661 2024-09-15T03:57:49.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 154618822661' 2024-09-15T03:57:49.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:57:49.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822659 -lt 154618822661 2024-09-15T03:57:49.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:57:50.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T03:57:50.638 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:57:51.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822659 -lt 154618822661 2024-09-15T03:57:51.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T03:57:52.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T03:57:52.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:57:52.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 154618822661 -lt 154618822661 2024-09-15T03:57:52.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:57:52.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-163208757252 2024-09-15T03:57:52.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:57:52.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T03:57:52.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-163208757252 2024-09-15T03:57:52.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:57:52.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=163208757252 2024-09-15T03:57:52.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 163208757252' 2024-09-15T03:57:52.366 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 163208757252 2024-09-15T03:57:52.366 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T03:57:52.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 163208757252 -lt 163208757252 2024-09-15T03:57:52.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:57:52.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:52.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:53.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:57:53.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:53.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:53.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:57:53.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:53.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:53.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:57:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:54.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:54.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:57:54.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:54.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:54.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:54.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:57:54.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:57:54.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:57:55.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:57:55.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:57:55.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:57:55.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:57:55.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:57:55.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:57:55.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:55.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:56.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:57:56.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:56.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:56.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:56.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:57:56.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:57:56.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:57:56.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:57:56.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:57:56.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:57:56.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:57:56.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:57.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:57.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:57.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:57.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:57:57.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:57.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:57.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:57:58.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:57:58.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:57:58.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:57:58.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:57:58.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:57:58.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:57:58.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:57:58.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:57:59.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:57:59.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:57:59.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:57:59.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:57:59.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:57:59.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:57:59.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:57:59.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:57:59.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:57:59.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:57:59.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:57:59.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:00.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:00.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:00.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:00.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:00.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:00.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:00.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:58:00.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:00.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:58:02.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:02.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:02.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:02.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:02.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:02.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:02.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:02.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:02.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:02.807 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:02.807 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:02.807 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:03.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:03.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:03.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:03.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:03.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:58:03.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:03.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:58:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:06.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:06.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:06.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:06.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:06.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:06.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:07.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:07.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:07.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:07.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:07.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:07.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:58:14.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:14.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:14.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:15.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:15.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:15.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:15.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:15.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:15.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:15.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:15.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:15.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:16.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:16.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:16.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:58:16.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:16.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:23.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:23.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:23.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:23.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:23.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:23.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:23.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:24.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:24.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:24.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:24.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:24.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:24.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:24.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:58:24.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:58:25.196 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "929cfa2f-3ff9-400d-bd9e-4c8489753f32", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:58:25.188824+0000", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:58:25.197 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:58:25.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.199 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.200 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:58:25.201 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:54:47.813717Z", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:54:47.813717Z", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:58:25.202 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:58:25.203 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:58:25.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 42, 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:54:48.602720+0000", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:57:42.873336+0000", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:57:40.693981+0000", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:57:42.689325+0000", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:58:25.205 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 10, 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:58:25.206 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:56:33.733709+0000", 2024-09-15T03:58:25.207 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 16389, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 3, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 1, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T03:58:25.208 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "29", 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:58:25.209 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "pool", 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 1, 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 29, 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [ 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "snapid": 1, 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "stamp": "2024-09-15T03:57:18.468335+0000", 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "snap1" 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:58:25.210 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:58:25.211 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "myprofile", 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 8192, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:58:25.212 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "14218549-25fd-49a0-801f-6454193e2a35", 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 5, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 28, 2024-09-15T03:58:25.213 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 36, 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 41, 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 30, 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:58:25.214 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.215 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.216 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83287 2024-09-15T03:58:25.217 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/83287", 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/83287", 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/83287", 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/83287", 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:58:25.218 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "ee84699d-c2a4-4409-8483-0497fc15a0ca", 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 25, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 31, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.219 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:58:25.220 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:58:25.221 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.222 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/76922", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/76922", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/76922", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/76922", 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:58:25.223 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4", 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 15, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 28, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 38, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 0, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 32, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:58:25.224 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.225 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:58:25.226 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:58:25.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 83865 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/83865", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/83865", 2024-09-15T03:58:25.228 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/83865", 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/83865", 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:57:29.394392+0000", 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:58:25.229 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:55:21.372149+0000", 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:57:29.894221+0000", 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:55:55.390360+0000", 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:57:30.397746+0000", 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:56:29.278893+0000", 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:58:25.231 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "myprofile": { 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-device-class": "", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-failure-domain": "osd", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-root": "default", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-15T03:58:25.232 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "1", 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van", 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "w": "8" 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:58:25.233 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:58:25.234 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/83287,v1:127.0.0.1:6805/83287]", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:58:25.236 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.237 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:54:54.296769Z", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/83287,v1:127.0.0.1:6803/83287]", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/83287,v1:127.0.0.1:6809/83287]", 2024-09-15T03:58:25.238 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/83287,v1:127.0.0.1:6807/83287]", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:58:25.239 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/76922,v1:127.0.0.1:6813/76922]", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:58:25.240 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:55:28.714967Z", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:58:25.241 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/76922,v1:127.0.0.1:6811/76922]", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/76922,v1:127.0.0.1:6817/76922]", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/76922,v1:127.0.0.1:6815/76922]", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:58:25.242 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/83865,v1:127.0.0.1:6813/83865]", 2024-09-15T03:58:25.243 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:58:25.244 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:56:02.156804Z", 2024-09-15T03:58:25.245 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/83865,v1:127.0.0.1:6811/83865]", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/83865,v1:127.0.0.1:6817/83865]", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/83865,v1:127.0.0.1:6815/83865]", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:58:25.246 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:58:25.247 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [] 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 36 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 38 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:58:25.248 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 42, 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.249 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.250 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.251 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:58:25.252 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:58:25.253 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.254 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.255 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:58:25.256 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.257 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:58:25.258 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:58:25.259 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 1, 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "test", 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 3, 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "set_chooseleaf_tries", 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 5 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.261 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "set_choose_tries", 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 100 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_indep", 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.262 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:58:25.263 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:58:25.264 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:58:25.264 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:58:25.264 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:58:25.264 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:58:25.264 INFO:tasks.workunit.client.0.smithi138.stderr:report 3125249612 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:58:25.265 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:58:25.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:58:25.267 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 1, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48454, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 101, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 1, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 303, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:58:25.268 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 101, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 101, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:58:25.269 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:58:25.270 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:58:25.271 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 18, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 18, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 3, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 3, 2024-09-15T03:58:25.272 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894656, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2140, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892469, 2024-09-15T03:58:25.273 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678144, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206419456, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2191360, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1535553, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 913888730 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:58:25.274 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:58:25.275 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 894656, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2140, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 10, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 892469, 2024-09-15T03:58:25.276 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313678144, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321206419456, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2191360, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1535553, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 10790, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 913888730 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:58:25.277 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:58:25.278 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48454, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 101, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 1, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 303, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 101, 2024-09-15T03:58:25.279 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 101, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:58:25.280 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:58:25.281 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1241088, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1241088, 2024-09-15T03:58:25.282 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 18, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 18, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 3, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 3, 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:58:25.283 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 154618822668 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672988 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 163208757259 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:58:25.284 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2147483647, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:58:25.285 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:58:25.286 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 5, 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:57:30.981937+0000", 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:58:25.287 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 539, 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:58:25.288 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:58:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-09-15T03:58:25.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-09-15T03:58:26.492 INFO:tasks.workunit.client.0.smithi138.stderr:PG '1.0' not found 2024-09-15T03:58:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-15T03:58:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-09-15T03:58:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:58:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:58:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:58:27.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:58:27.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:58:27.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:58:27.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-09-15T03:58:27.012 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:58:27.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T03:58:27.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:58:27.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-09-15T03:58:27.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T03:58:27.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:58:27.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:58:27.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:58:27.043 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:27.042+0000 7ff3f8d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:27.061 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:27.061+0000 7ff3f8d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:27.062 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:27.062+0000 7ff3f8d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:58:27.408 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:58:27.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:58:27.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:58:27.585 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:27.585+0000 7ff3f8d21440 -1 Falling back to public interface 2024-09-15T03:58:27.806 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 36 up_thru 41 down_at 30 last_clean_interval [5,28) [v2:127.0.0.1:6802/83287,v1:127.0.0.1:6803/83287] [v2:127.0.0.1:6804/83287,v1:127.0.0.1:6805/83287] exists,up 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:58:27.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:58:27.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:58:27.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:58:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:58:27.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-09-15T03:58:27.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-09-15T03:58:29.282 INFO:tasks.workunit.client.0.smithi138.stderr:PG '1.0' not found 2024-09-15T03:58:29.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:58:29.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:58:29.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-09-15T03:58:29.801 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T03:58:29.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:58:29.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:58:29.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:29.828+0000 7fda1cf1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:29.830 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:29.830+0000 7fda1cf1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:29.831 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:29.831+0000 7fda1cf1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:58:30.066 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:30.065+0000 7ff3f8d21440 -1 osd.0 42 log_to_monitors true 2024-09-15T03:58:30.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:58:30.163 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:58:30.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:58:30.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:58:30.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:58:30.352 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:30.352+0000 7fda1cf1f440 -1 Falling back to public interface 2024-09-15T03:58:30.570 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,28) [v2:127.0.0.1:6810/83865,v1:127.0.0.1:6811/83865] [v2:127.0.0.1:6812/83865,v1:127.0.0.1:6813/83865] exists,up f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-09-15T03:58:30.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T03:58:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T03:58:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T03:58:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-09-15T03:58:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-09-15T03:58:32.134 INFO:tasks.workunit.client.0.smithi138.stderr:PG '1.0' not found 2024-09-15T03:58:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-15T03:58:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T03:58:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-09-15T03:58:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T03:58:32.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T03:58:32.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T03:58:32.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T03:58:32.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T03:58:32.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T03:58:32.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T03:58:32.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T03:58:32.754 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:58:32.753+0000 7fda1cf1f440 -1 osd.2 42 log_to_monitors true 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:58:32.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T03:58:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=193273528323 2024-09-15T03:58:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 193273528323 2024-09-15T03:58:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-193273528323' 2024-09-15T03:58:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:58:32.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T03:58:33.092 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-15T03:58:33.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:58:33.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:58:33.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:58:33.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T03:58:33.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T03:58:33.252 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.2 2024-09-15T03:58:33.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T03:58:33.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T03:58:33.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T03:58:33.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T03:58:33.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-193273528323 2024-09-15T03:58:33.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T03:58:33.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T03:58:33.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-193273528323 2024-09-15T03:58:33.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T03:58:33.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=193273528323 2024-09-15T03:58:33.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 193273528323' 2024-09-15T03:58:33.257 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 193273528323 2024-09-15T03:58:33.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T03:58:33.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 193273528323 -lt 193273528323 2024-09-15T03:58:33.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T03:58:33.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:33.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:34.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T03:58:34.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:34.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:34.607 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:34.607 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:34.607 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:35.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:35.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:35.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:35.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:35.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:35.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:35.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:35.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:36.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:36.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:36.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:36.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:36.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-09-15T03:58:36.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:36.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T03:58:36.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:36.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:36.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:36.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:36.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:36.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:36.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:36.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:37.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:37.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:37.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:37.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:37.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:38.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:38.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:38.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-09-15T03:58:38.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:38.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:38.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:38.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:38.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:38.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:38.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:39.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:39.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:39.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:39.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-09-15T03:58:39.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:39.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T03:58:40.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:40.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:40.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:40.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:40.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:40.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:41.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:41.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:41.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:41.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:41.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-09-15T03:58:41.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:41.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T03:58:43.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:43.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:43.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:43.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:43.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:43.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:43.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:43.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:43.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:43.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:43.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:43.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:44.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:44.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:44.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:44.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:44.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:44.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-09-15T03:58:44.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:44.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T03:58:48.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:48.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:48.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:48.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:48.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:48.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:48.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:48.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:48.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:48.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:48.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:48.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:48.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:48.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:48.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:48.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-09-15T03:58:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T03:58:55.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:58:55.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:58:55.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:58:56.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:58:56.155 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:58:56.155 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:58:56.155 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:58:56.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:58:56.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:58:57.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:58:57.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:58:57.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-09-15T03:58:57.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T03:58:57.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T03:59:04.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T03:59:04.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T03:59:04.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T03:59:04.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T03:59:04.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T03:59:04.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T03:59:04.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T03:59:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T03:59:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T03:59:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T03:59:05.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T03:59:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T03:59:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T03:59:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-09-15T03:59:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "929cfa2f-3ff9-400d-bd9e-4c8489753f32", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T03:59:06.171833+0000", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_WARN", 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "OSDMAP_FLAGS": { 2024-09-15T03:59:06.173 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "nodown,noout,noin flag(s) set", 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 17 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "OSD_DOWN": { 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 osds down", 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.174 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "osd.1 (root=default,host=smithi138) is down" 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:59:06.175 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 pool(s) do not have an application enabled", 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T03:59:06.176 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'test'" 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T03:59:06.177 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:54:47.813717Z", 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:54:47.813717Z", 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T03:59:06.178 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T03:59:06.179 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.180 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115", 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7115/0", 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7115/0", 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.181 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 51, 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "1fff9473-5a64-4d6b-8ef2-a9211b515210", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T03:54:48.602720+0000", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:58:33.776977+0000", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T03:58:33.263818+0000", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T03:57:42.689325+0000", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5800384, 2024-09-15T03:59:06.182 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "nodown", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "noin", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "noout", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 12, 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T03:59:06.183 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 1, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 3, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "jewel", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "test", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T03:56:33.733709+0000", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 16389, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 3, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T03:59:06.184 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T03:59:06.185 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "29", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "pool", 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 1, 2024-09-15T03:59:06.186 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 29, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [ 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "snapid": 1, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "stamp": "2024-09-15T03:57:18.468335+0000", 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "snap1" 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T03:59:06.187 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "myprofile", 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T03:59:06.188 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 8192, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "14218549-25fd-49a0-801f-6454193e2a35", 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 36, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 42, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 45, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 50, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 44, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:59:06.190 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.191 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.192 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.193 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 85861 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/85861", 2024-09-15T03:59:06.194 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/85861", 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/85861", 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/85861", 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "ee84699d-c2a4-4409-8483-0497fc15a0ca", 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 0, 2024-09-15T03:59:06.195 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 10, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 25, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 31, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.196 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.197 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.198 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.199 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 76922 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/76922", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/76922", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/76922", 2024-09-15T03:59:06.200 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/76922", 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "exists" 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4", 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 38, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 42, 2024-09-15T03:59:06.201 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 49, 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 0, 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 48, 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.202 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.203 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T03:59:06.204 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 86724 2024-09-15T03:59:06.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/86724", 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/86724", 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/86724", 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/86724", 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T03:59:06.206 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:58:30.359603+0000", 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:55:21.372149+0000", 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:57:29.894221+0000", 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:59:06.207 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:55:55.390360+0000", 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "2024-09-15T03:58:33.088553+0000", 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T03:56:29.278893+0000", 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T03:59:06.208 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [], 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:59:06.209 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "myprofile": { 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-device-class": "", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-failure-domain": "osd", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "crush-root": "default", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "jerasure-per-chunk-alignment": "false", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "1", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van", 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "w": "8" 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T03:59:06.210 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.211 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/85861,v1:127.0.0.1:6805/85861]", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:59:06.212 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:54:54.296769Z", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:59:06.213 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/85861,v1:127.0.0.1:6803/85861]", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/85861,v1:127.0.0.1:6809/85861]", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/85861,v1:127.0.0.1:6807/85861]", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:59:06.214 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/0", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/76922,v1:127.0.0.1:6813/76922]", 2024-09-15T03:59:06.215 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:59:06.216 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:55:28.714967Z", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:59:06.217 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/76922,v1:127.0.0.1:6811/76922]", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/76922,v1:127.0.0.1:6817/76922]", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/76922,v1:127.0.0.1:6815/76922]", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:59:06.218 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/1", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/86724,v1:127.0.0.1:6813/86724]", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T03:59:06.219 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.220 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T03:56:02.156804Z", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/86724,v1:127.0.0.1:6811/86724]", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/86724,v1:127.0.0.1:6817/86724]", 2024-09-15T03:59:06.221 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/86724,v1:127.0.0.1:6815/86724]", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/divergent-priors/2", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.222 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [] 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 46 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 50 2024-09-15T03:59:06.223 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 51, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:59:06.224 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.225 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T03:59:06.226 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T03:59:06.227 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T03:59:06.228 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.229 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:59:06.230 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:59:06.231 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.232 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 19206, 2024-09-15T03:59:06.233 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T03:59:06.236 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.237 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:59:06.238 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 1, 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "test", 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 3, 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "set_chooseleaf_tries", 2024-09-15T03:59:06.239 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 5 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "set_choose_tries", 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 100 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_indep", 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T03:59:06.240 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T03:59:06.241 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T03:59:06.242 INFO:tasks.workunit.client.0.smithi138.stderr:report 4209960293 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 1, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T03:59:06.243 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T03:59:06.244 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 5, 2024-09-15T03:59:06.245 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 3 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 1, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 3, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48454, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 101, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 1, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 303, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:59:06.246 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 101, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 101, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:59:06.247 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:59:06.248 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:59:06.249 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 18, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 18, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 3, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 3, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:59:06.250 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895720, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2284, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 16, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 893359, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677080, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205329920, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2338816, 2024-09-15T03:59:06.251 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1595193, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 17014, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914800010 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:59:06.252 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T03:59:06.253 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 3, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 3, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 3, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 3, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 314572800, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 895720, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 2284, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 16, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 893359, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 313677080, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 322122547200, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 321205329920, 2024-09-15T03:59:06.254 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 2338816, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1595193, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 17014, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 914800010 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 0, 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T03:59:06.255 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [], 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 1 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T03:59:06.256 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 48454, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 101, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 1, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 303, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 101, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 101, 2024-09-15T03:59:06.257 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 101, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 101, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T03:59:06.258 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T03:59:06.259 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1241088, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 1241088, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T03:59:06.260 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 18, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 18, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 3, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 3, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 193273528329 2024-09-15T03:59:06.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 42949672988 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 210453397511 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+undersized+degraded", 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T03:59:06.262 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2147483647, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.263 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 1, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 0, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 0, 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.264 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 5, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T03:57:30.981937+0000", 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 644, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T03:59:06.265 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T03:59:06.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T03:59:06.266 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T03:59:06.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T03:59:06.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-09-15T03:59:06.266 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery 2024-09-15T03:59:06.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:06.566 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:06.566 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 51:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:06.566 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:06.566 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:06.567 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:06.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.0.asok 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T03:59:06.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.0.asok set_recovery_delay 100000 2024-09-15T03:59:06.653 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:59:06.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.2.asok 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T03:59:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.2.asok set_recovery_delay 100000 2024-09-15T03:59:06.735 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:59:06.742 INFO:tasks.workunit.client.0.smithi138.stdout:revive divergent 1 2024-09-15T03:59:06.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-09-15T03:59:06.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:07.075 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:07.075 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 51:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:07.076 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:07.076 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:07.076 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:07.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-09-15T03:59:07.683 INFO:tasks.workunit.client.0.smithi138.stderr:noup is set 2024-09-15T03:59:07.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-15T03:59:07.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:59:07.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:59:07.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:59:07.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:59:07.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:59:07.701 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:59:07.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:59:07.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:59:07.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:59:07.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:59:07.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:59:07.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:59:07.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:59:07.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:07.726+0000 7f025f209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:07.729 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:07.728+0000 7f025f209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:07.730 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:07.729+0000 7f025f209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:08.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-09-15T03:59:08.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-09-15T03:59:08.252 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:08.252+0000 7f025f209440 -1 Falling back to public interface 2024-09-15T03:59:10.764 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:10.764+0000 7f025f209440 -1 osd.1 29 log_to_monitors true 2024-09-15T03:59:13.115 INFO:tasks.workunit.client.0.smithi138.stdout:delay recovery divergent 2024-09-15T03:59:13.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-09-15T03:59:13.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:13.463 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:13.463 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 52:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:13.463 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:13.463 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:13.463 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:13.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.32635/ceph-osd.1.asok 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-09-15T03:59:13.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.32635/ceph-osd.1.asok set_recovery_delay 100000 2024-09-15T03:59:13.549 INFO:tasks.workunit.client.0.smithi138.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-09-15T03:59:13.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-09-15T03:59:14.163 INFO:tasks.workunit.client.0.smithi138.stderr:noup is unset 2024-09-15T03:59:14.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:14.175 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:14.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:59:14.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:14.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:14.724 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 45 up_thru 50 down_at 44 last_clean_interval [36,42) [v2:127.0.0.1:6802/85861,v1:127.0.0.1:6803/85861] [v2:127.0.0.1:6804/85861,v1:127.0.0.1:6805/85861] exists,up 14218549-25fd-49a0-801f-6454193e2a35 2024-09-15T03:59:14.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:14.726 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:14.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:59:14.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:14.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:59:15.077 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/88573,v1:127.0.0.1:6819/88573] [v2:127.0.0.1:6820/88573,v1:127.0.0.1:6821/88573] exists,up ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T03:59:15.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T03:59:15.440 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 49 up_thru 0 down_at 48 last_clean_interval [38,42) [v2:127.0.0.1:6810/86724,v1:127.0.0.1:6811/86724] [v2:127.0.0.1:6812/86724,v1:127.0.0.1:6813/86724] exists,up f73cb9ad-6a19-4f1f-9eb9-1eee7e4439c4 2024-09-15T03:59:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:59:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:59:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:59:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:15.787 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:15.787 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 52:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:15.788 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:15.788 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:15.788 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:15.798 INFO:tasks.workunit.client.0.smithi138.stdout:wait for peering 2024-09-15T03:59:15.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-09-15T03:59:15.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:16.134 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:16.134 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 52:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:16.134 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:16.134 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:16.134 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:16.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-09-15T03:59:16.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-09-15T03:59:16.265 INFO:tasks.workunit.client.0.smithi138.stdout:killing divergent 1 2024-09-15T03:59:16.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:16.597 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:16.598 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 0 101 0 0 48454 0 0 18 84 18 active+undersized+degraded 2024-09-15T03:58:34.027073+0000 42'102 52:146 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:21:02.723797+0000 0 0 2024-09-15T03:59:16.598 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:16.598 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:16.598 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:16.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-09-15T03:59:16.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:59:16.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:59:16.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:59:16.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:59:16.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:59:16.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:59:16.714 INFO:tasks.workunit.client.0.smithi138.stdout:reviving divergent 1 2024-09-15T03:59:16.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-09-15T03:59:16.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:17.049 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:17.049 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 101 4 4 0 0 48454 0 0 12 90 12 active+recovering+degraded 2024-09-15T03:59:14.867581+0000 42'102 57:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:50:47.426637+0000 0 0 2024-09-15T03:59:17.049 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:17.049 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:17.049 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:17.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-09-15T03:59:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-09-15T03:59:17.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-09-15T03:59:17.064 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1fff9473-5a64-4d6b-8ef2-a9211b515210 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.32635/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T03:59:17.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T03:59:17.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T03:59:17.092 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:17.091+0000 7f4cce84a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:17.094 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:17.093+0000 7f4cce84a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:17.095 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:17.094+0000 7f4cce84a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:17.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T03:59:17.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:59:17.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T03:59:17.422 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:17.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T03:59:17.608 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:17.608+0000 7f4cce84a440 -1 Falling back to public interface 2024-09-15T03:59:17.782 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 56 up_thru 56 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/88573,v1:127.0.0.1:6819/88573] [v2:127.0.0.1:6820/88573,v1:127.0.0.1:6821/88573] exists,up ee84699d-c2a4-4409-8483-0497fc15a0ca 2024-09-15T03:59:17.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T03:59:17.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T03:59:17.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T03:59:17.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-09-15T03:59:20.533 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:20.532+0000 7f4cce84a440 -1 osd.1 57 log_to_monitors true 2024-09-15T03:59:37.785 INFO:tasks.workunit.client.0.smithi138.stdout:allowing recovery 2024-09-15T03:59:37.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-09-15T03:59:37.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:38.128 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:38.128 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-15T03:59:22.284364+0000 57'103 62:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:42:03.255885+0000 0 0 2024-09-15T03:59:38.128 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:38.128 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:38.129 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:38.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T03:59:38.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T03:59:38.243 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:59:38.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T03:59:38.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T03:59:38.355 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:59:38.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-09-15T03:59:38.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T03:59:38.467 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T03:59:38.475 INFO:tasks.workunit.client.0.smithi138.stdout:reading divergent objects 2024-09-15T03:59:38.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-09-15T03:59:38.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-09-15T03:59:38.848 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T03:59:38.849 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 102 0 0 0 0 48937 0 0 13 90 13 active+clean 2024-09-15T03:59:22.284364+0000 57'103 62:155 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T03:56:33.792907+0000 0'0 2024-09-15T03:56:33.792907+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:42:03.255885+0000 0 0 2024-09-15T03:59:38.849 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T03:59:38.849 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T03:59:38.849 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T03:59:38.859 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-09-15T03:59:38.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-09-15T03:59:38.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:59:38.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-09-15T03:59:38.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:59:38.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-09-15T03:59:38.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:59:38.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-09-15T03:59:38.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-09-15T03:59:38.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-09-15T03:59:38.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-09-15T03:59:38.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-15T03:59:38.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-15T03:57:11.483253+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: keeping 29'103 (28'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-09-15T03:57:11.483253+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 28'99 first_divergent_update: 29'103 last_divergent_update: 29'103 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-15T03:57:18.653127+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: keeping 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-15T03:57:18.653127+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 28'99 first_divergent_update: 29'104 last_divergent_update: 29'104 2024-09-15T03:59:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 29'104 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 29'104 (28'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-09-15T03:57:18.653127+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-15T03:57:17.461644+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-15T03:57:18.468363+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-09-15T03:57:17.461644+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 20 _merge_object_divergent_entries: keeping 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-15T03:57:18.468363+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.001 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'102 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'104 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'102 (28'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-09-15T03:57:18.468363+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stdout:td/divergent-priors/osd.1.log:2024-09-15T03:59:14.674+0000 7f02421c7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-09-15T03:59:39.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-09-15T03:59:39.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2024-09-15T03:59:39.009 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T03:59:39.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-09-15T03:59:39.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-09-15T03:59:39.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T03:59:39.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T03:59:39.489 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T03:59:39.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-09-15T03:59:39.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:59:39.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:59:39.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:59:39.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:59:39.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:59:44.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:59:44.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:59:44.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:59:44.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:59:44.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:59:44.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:59:44.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:59:44.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:59:44.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:59:44.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:59:44.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:59:44.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:59:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:59:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:59:44.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:44.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:59:44.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:59:44.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:59:44.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:59:44.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:44.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:44.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/divergent-priors 0 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T03:59:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:59:44.859 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:59:44.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:59:44.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:59:44.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:59:44.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:59:44.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:59:44.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:44.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:59:44.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:59:44.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:44.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:59:44.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T03:59:44.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-09-15T03:59:44.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:59:44.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:44.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.32635 2024-09-15T03:59:44.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.32635 2024-09-15T03:59:44.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:59:44.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:59:44.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T03:59:44.869 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T03:59:44.869 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T03:59:44.938 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-09-15T03:59:44.938 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-09-15T03:59:45.003 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T03:59:45.009 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T03:59:45.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/ec-error-rollforward 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-09-15T03:59:45.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none ' 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-09-15T03:59:45.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-09-15T03:59:45.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T03:59:45.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T03:59:45.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T03:59:45.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T03:59:45.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T03:59:45.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T03:59:45.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T03:59:45.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:45.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T03:59:45.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T03:59:45.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T03:59:45.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T03:59:45.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T03:59:45.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-15T03:59:45.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T03:59:45.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:45.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:45.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90131 2024-09-15T03:59:45.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T03:59:45.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T03:59:45.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-09-15T03:59:45.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T03:59:45.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:45.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:45.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.90131 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-09-15T03:59:45.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-09-15T03:59:45.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T03:59:45.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T03:59:45.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T03:59:45.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T03:59:45.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:59:45.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:59:45.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T03:59:45.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90131/ceph-mon.a.asok 2024-09-15T03:59:45.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:59:45.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.90131/ceph-mon.a.asok config get fsid 2024-09-15T03:59:45.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:45.716 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.90131/ceph-mon.a.asok 2024-09-15T03:59:45.717 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T03:59:45.717 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.90131/ceph-mon.a.asok config get mon_host 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-09-15T03:59:45.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:46.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T03:59:46.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:59:46.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T03:59:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T03:59:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T03:59:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f013e18-99d2-4a60-b8df-27595be420cd 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8f013e18-99d2-4a60-b8df-27595be420cd' 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 8f013e18-99d2-4a60-b8df-27595be420cd 2024-09-15T03:59:46.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T03:59:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyW+ZmjFqfJxAAwwq3dc2mOrbDpxODRk1wDQ== 2024-09-15T03:59:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyW+ZmjFqfJxAAwwq3dc2mOrbDpxODRk1wDQ=="}' 2024-09-15T03:59:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f013e18-99d2-4a60-b8df-27595be420cd -i td/ec-error-rollforward/0/new.json 2024-09-15T03:59:47.052 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:47.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-09-15T03:59:47.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyW+ZmjFqfJxAAwwq3dc2mOrbDpxODRk1wDQ== --osd-uuid 8f013e18-99d2-4a60-b8df-27595be420cd 2024-09-15T03:59:47.084 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.083+0000 7f06f8520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:47.107 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.106+0000 7f06f8520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:47.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.107+0000 7f06f8520440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:47.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.108+0000 7f06f8520440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-15T03:59:47.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.108+0000 7f06f8520440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-15T03:59:47.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.108+0000 7f06f8520440 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-09-15T03:59:47.147 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:47.146+0000 7f06f8520440 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-09-15T03:59:52.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-09-15T03:59:52.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T03:59:52.472 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T03:59:52.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T03:59:52.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T03:59:52.951 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T03:59:52.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T03:59:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T03:59:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T03:59:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T03:59:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T03:59:52.971 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:52.971+0000 7fe87511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:52.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:52.973+0000 7fe87511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:52.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:52.974+0000 7fe87511d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T03:59:53.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T03:59:53.302 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T03:59:53.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:53.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:53.491 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:53.491+0000 7fe87511d440 -1 Falling back to public interface 2024-09-15T03:59:53.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:59:54.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:59:54.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:54.655 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T03:59:54.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T03:59:54.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:54.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:55.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:59:55.951 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T03:59:55.951+0000 7fe87511d440 -1 osd.0 0 log_to_monitors true 2024-09-15T03:59:56.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:59:56.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:56.008 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T03:59:56.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T03:59:56.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:56.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:56.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:59:57.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:59:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:57.514 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T03:59:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T03:59:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:57.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T03:59:58.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T03:59:58.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T03:59:58.966 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T03:59:58.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T03:59:58.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T03:59:58.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T03:59:59.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:00.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:00.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:00.364 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:00:00.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:00:00.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:00.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:00.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:01.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:01.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:01.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:00:01.807 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:00:01.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:01.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:02.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:03.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:03.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:03.211 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:00:03.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:00:03.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:03.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:03.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:04.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:04.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:04.617 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:00:04.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:00:04.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:04.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:05.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:06.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:06.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:06.007 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:00:06.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:00:06.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:06.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:06.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:07.394 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:00:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:00:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:08.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:09.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:09.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:09.431 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:00:09.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:00:09.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:09.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:10.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:11.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:00:11.006 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:00:11.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:11.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:11.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:12.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:12.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:12.818 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:00:12.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:00:12.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:12.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:13.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:14.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:14.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:14.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:00:14.908 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:00:14.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:14.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:15.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:16.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:16.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:16.348 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:00:16.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:00:16.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:16.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:16.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:17.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:17.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:17.858 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:00:17.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:00:17.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:17.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:18.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:19.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:19.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:19.373 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:00:19.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:00:19.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:19.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:00:19.723 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91222,v1:127.0.0.1:6803/91222] [v2:127.0.0.1:6804/91222,v1:127.0.0.1:6805/91222] exists,up 8f013e18-99d2-4a60-b8df-27595be420cd 2024-09-15T04:00:19.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:00:19.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:00:19.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:00:19.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:00:19.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed75363c-d44c-4c86-9004-abb9febf8626 2024-09-15T04:00:19.730 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 ed75363c-d44c-4c86-9004-abb9febf8626 2024-09-15T04:00:19.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ed75363c-d44c-4c86-9004-abb9febf8626' 2024-09-15T04:00:19.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:00:19.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDTW+ZmIeAMLBAAldEQL/IflqNAbs+kXxUXcA== 2024-09-15T04:00:19.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDTW+ZmIeAMLBAAldEQL/IflqNAbs+kXxUXcA=="}' 2024-09-15T04:00:19.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed75363c-d44c-4c86-9004-abb9febf8626 -i td/ec-error-rollforward/1/new.json 2024-09-15T04:00:20.201 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:00:20.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-09-15T04:00:20.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDTW+ZmIeAMLBAAldEQL/IflqNAbs+kXxUXcA== --osd-uuid ed75363c-d44c-4c86-9004-abb9febf8626 2024-09-15T04:00:20.235 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.234+0000 7f027be0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:20.236 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.236+0000 7f027be0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:20.237 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.237+0000 7f027be0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:20.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.237+0000 7f027be0c440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-15T04:00:20.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.237+0000 7f027be0c440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-15T04:00:20.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.237+0000 7f027be0c440 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-09-15T04:00:20.283 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:20.282+0000 7f027be0c440 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-09-15T04:00:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-09-15T04:00:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:00:26.203 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:00:26.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:00:26.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:00:26.684 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:00:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:00:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:00:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:00:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:00:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:00:26.704 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:26.703+0000 7f2890921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:26.706 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:26.705+0000 7f2890921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:26.706+0000 7f2890921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:27.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:00:27.031 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:00:27.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:27.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:27.223 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:27.222+0000 7f2890921440 -1 Falling back to public interface 2024-09-15T04:00:27.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:28.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:28.391 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:00:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:00:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:28.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:29.705 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:29.704+0000 7f2890921440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:00:29.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:29.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:29.742 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:00:29.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:00:29.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:29.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:30.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:31.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:31.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:31.507 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:00:31.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:00:31.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:31.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:31.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:32.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:32.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:32.896 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:00:32.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:00:32.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:32.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:33.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:34.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:34.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:34.309 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:00:34.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:00:34.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:34.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:34.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:35.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:35.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:35.964 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:00:35.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:00:35.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:35.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:37.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:37.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:37.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:00:37.563 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:00:37.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:37.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:37.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:38.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:38.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:38.972 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:00:38.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:00:38.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:38.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:39.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:40.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:40.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:40.523 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:00:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:00:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:40.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:41.984 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:00:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:00:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:42.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:43.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:43.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:43.492 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:00:43.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:00:43.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:43.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:44.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:45.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:45.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:45.115 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:00:45.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:00:45.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:45.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:45.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:46.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:46.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:46.879 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:00:46.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:00:46.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:46.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:47.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:48.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:48.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:48.590 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:00:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:00:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:49.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:50.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:50.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:50.543 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:00:50.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:00:50.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:50.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:51.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:52.714 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:00:52.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:00:52.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:52.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:53.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:00:54.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:00:54.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:00:54.234 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:00:54.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:00:54.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:00:54.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:00:54.594 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92977,v1:127.0.0.1:6811/92977] [v2:127.0.0.1:6812/92977,v1:127.0.0.1:6813/92977] exists,up ed75363c-d44c-4c86-9004-abb9febf8626 2024-09-15T04:00:54.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:00:54.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:00:54.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:00:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=75d51319-8b7d-4166-8239-6f38e7247171 2024-09-15T04:00:54.601 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 75d51319-8b7d-4166-8239-6f38e7247171 2024-09-15T04:00:54.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 75d51319-8b7d-4166-8239-6f38e7247171' 2024-09-15T04:00:54.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:00:54.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2W+ZmiL5jJBAAyFox4t7Frl1QEAe0D5SO2A== 2024-09-15T04:00:54.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2W+ZmiL5jJBAAyFox4t7Frl1QEAe0D5SO2A=="}' 2024-09-15T04:00:54.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 75d51319-8b7d-4166-8239-6f38e7247171 -i td/ec-error-rollforward/2/new.json 2024-09-15T04:00:55.079 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:00:55.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-09-15T04:00:55.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2W+ZmiL5jJBAAyFox4t7Frl1QEAe0D5SO2A== --osd-uuid 75d51319-8b7d-4166-8239-6f38e7247171 2024-09-15T04:00:55.113 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.113+0000 7fe25608b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:55.115 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.115+0000 7fe25608b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:55.116 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.116+0000 7fe25608b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:00:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.116+0000 7fe25608b440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-15T04:00:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.116+0000 7fe25608b440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-15T04:00:55.117 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.116+0000 7fe25608b440 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-09-15T04:00:55.170 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:00:55.170+0000 7fe25608b440 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-09-15T04:01:01.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-09-15T04:01:01.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:01:01.045 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:01:01.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:01:01.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:01:01.526 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:01:01.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:01:01.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:01:01.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:01:01.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:01:01.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:01:01.548 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:01.547+0000 7fa9c604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:01.549 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:01.549+0000 7fa9c604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:01.550 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:01.550+0000 7fa9c604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:01.878 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:01:01.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:01:01.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:01.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:02.065 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:02.065+0000 7fa9c604f440 -1 Falling back to public interface 2024-09-15T04:01:02.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:03.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:03.231 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:01:03.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:03.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:01:03.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:03.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:03.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:04.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:04.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:04.577 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:01:04.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:01:04.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:04.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:04.633 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:04.633+0000 7fa9c604f440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:01:04.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:05.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:05.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:05.950 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:01:05.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:01:05.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:05.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:06.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:07.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:07.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:07.516 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:01:07.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:01:07.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:07.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:07.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:08.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:08.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:08.915 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:01:08.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:01:08.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:08.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:09.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:10.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:10.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:10.282 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:01:10.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:01:10.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:10.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:10.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:01:11.763 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:01:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:12.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:13.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:13.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:13.133 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:01:13.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:01:13.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:13.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:13.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:14.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:14.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:14.580 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:01:14.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:01:14.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:14.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:15.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:16.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:16.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:16.030 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:01:16.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:01:16.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:16.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:16.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:17.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:17.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:17.603 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:01:17.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:01:17.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:17.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:18.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:19.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:19.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:19.366 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:01:19.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:01:19.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:19.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:20.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:21.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:21.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:21.355 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:01:21.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:01:21.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:21.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:21.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:22.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:22.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:22.914 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:01:22.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:01:22.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:22.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:23.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:24.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:24.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:24.909 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:01:24.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:01:24.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:24.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:25.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:26.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:26.618 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:01:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:01:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:27.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:28.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:28.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:28.882 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:01:28.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:01:28.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:28.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:01:29.240 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/94749,v1:127.0.0.1:6819/94749] [v2:127.0.0.1:6820/94749,v1:127.0.0.1:6821/94749] exists,up 75d51319-8b7d-4166-8239-6f38e7247171 2024-09-15T04:01:29.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:01:29.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:01:29.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:01:29.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-09-15T04:01:29.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:01:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:01:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0e2cd43a-4a2b-4065-aa3c-7be7a727f47f 2024-09-15T04:01:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f' 2024-09-15T04:01:29.247 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f 2024-09-15T04:01:29.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:01:29.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZXOZmtaNADxAAwAgUvsS5I+Y3EiMHO3zGmg== 2024-09-15T04:01:29.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZXOZmtaNADxAAwAgUvsS5I+Y3EiMHO3zGmg=="}' 2024-09-15T04:01:29.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f -i td/ec-error-rollforward/3/new.json 2024-09-15T04:01:29.735 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:01:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-09-15T04:01:29.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZXOZmtaNADxAAwAgUvsS5I+Y3EiMHO3zGmg== --osd-uuid 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f 2024-09-15T04:01:29.769 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.769+0000 7f887f157440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:29.771 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.771+0000 7f887f157440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:29.772 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.771+0000 7f887f157440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:29.772 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.772+0000 7f887f157440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-15T04:01:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.772+0000 7f887f157440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-15T04:01:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.772+0000 7f887f157440 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-09-15T04:01:29.851 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:29.850+0000 7f887f157440 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-09-15T04:01:36.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-09-15T04:01:36.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:01:36.027 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T04:01:36.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T04:01:36.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:01:36.513 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:01:36.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T04:01:36.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:01:36.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:01:36.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:01:36.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:01:36.533 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:36.533+0000 7f2e97809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:36.535 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:36.535+0000 7f2e97809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:36.536 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:36.536+0000 7f2e97809440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:01:36.875 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:01:36.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:36.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:37.053 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:37.053+0000 7f2e97809440 -1 Falling back to public interface 2024-09-15T04:01:37.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:38.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:38.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:38.246 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:01:38.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:01:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:38.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:39.565 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:01:39.564+0000 7f2e97809440 -1 osd.3 0 log_to_monitors true 2024-09-15T04:01:39.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:39.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:01:39.603 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:01:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:41.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:41.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:41.232 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:01:41.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:01:41.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:41.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:41.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:42.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:42.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:42.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:01:42.654 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:01:42.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:42.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:43.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:44.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:44.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:44.140 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:01:44.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:01:44.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:44.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:44.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:45.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:45.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:45.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:01:45.612 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:01:45.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:45.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:45.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:46.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:46.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:46.979 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:01:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:01:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:46.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:47.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:48.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:48.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:48.468 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:01:48.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:01:48.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:48.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:48.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:49.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:49.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:49.990 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:01:49.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:01:49.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:49.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:50.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:51.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:51.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:51.465 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:01:51.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:01:51.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:51.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:51.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:52.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:52.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:52.862 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:01:52.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:01:52.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:52.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:53.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:54.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:54.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:54.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:01:54.553 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:01:54.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:54.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:55.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:56.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:56.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:56.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:01:56.696 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:01:56.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:56.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:57.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:01:58.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:01:58.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:01:58.466 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:01:58.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:01:58.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:01:58.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:01:59.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:00.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:00.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:00.301 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:02:00.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:02:00.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:00.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:02:01.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:02.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:02.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:02.302 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:02:02.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:02:02.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:02.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:02:03.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:04.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:04.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:04.108 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:02:04.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:02:04.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:04.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:02:04.462 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96503,v1:127.0.0.1:6827/96503] [v2:127.0.0.1:6828/96503,v1:127.0.0.1:6829/96503] exists,up 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f 2024-09-15T04:02:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-15T04:02:05.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-09-15T04:02:05.779 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'ec' already exists 2024-09-15T04:02:05.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-09-15T04:02:07.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-09-15T04:02:07.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 94749 2024-09-15T04:02:07.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=97484 2024-09-15T04:02:07.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-09-15T04:02:07.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 97505' 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 97506' 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 97507' 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-09-15T04:02:08.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-09-15T04:02:09.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-09-15T04:02:09.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:02:09.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:02:09.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:02:09.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:02:09.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:02:09.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:02:09.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 97484 97505 97506 97507 2024-09-15T04:02:09.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:02:09.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:09.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:02:09.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-09-15T04:02:09.239 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:02:09.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:02:09.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:02:09.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-09-15T04:02:09.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:02:09.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:02:09.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:02:09.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:02:09.260 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.259+0000 7f3d29121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:09.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.262+0000 7f3d29121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:09.263 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.263+0000 7f3d29121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:09.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:02:09.593 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:09.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:09.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:09.779 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.779+0000 7f3d29121440 -1 Falling back to public interface 2024-09-15T04:02:09.940 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/91222,v1:127.0.0.1:6803/91222] [v2:127.0.0.1:6804/91222,v1:127.0.0.1:6805/91222] exists,up 8f013e18-99d2-4a60-b8df-27595be420cd 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:02:09.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:02:09.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:02:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:02:09.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:02:09.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-09-15T04:02:09.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T04:02:09.946 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:02:09.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:02:09.962 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.962+0000 7f2ef7c12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:09.964 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.964+0000 7f2ef7c12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:09.965 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:09.965+0000 7f2ef7c12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:02:10.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:10.302 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:10.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:02:10.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:10.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:02:10.482 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:10.482+0000 7f2ef7c12440 -1 Falling back to public interface 2024-09-15T04:02:10.651 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 23 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/92977,v1:127.0.0.1:6811/92977] [v2:127.0.0.1:6812/92977,v1:127.0.0.1:6813/92977] exists,up ed75363c-d44c-4c86-9004-abb9febf8626 2024-09-15T04:02:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:02:10.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:02:10.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-09-15T04:02:10.657 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:02:10.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T04:02:10.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:02:10.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-09-15T04:02:10.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T04:02:10.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:02:10.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:02:10.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:02:10.675 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:10.674+0000 7f6731f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:10.676 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:10.676+0000 7f6731f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:10.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:10.677+0000 7f6731f21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:11.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T04:02:11.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:02:11.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:02:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:02:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:02:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:11.006 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:11.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:02:11.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:11.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:02:11.202 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:11.202+0000 7f6731f21440 -1 Falling back to public interface 2024-09-15T04:02:11.358 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/94749,v1:127.0.0.1:6819/94749] [v2:127.0.0.1:6820/94749,v1:127.0.0.1:6821/94749] exists,up 75d51319-8b7d-4166-8239-6f38e7247171 2024-09-15T04:02:11.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:11.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:11.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:11.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-09-15T04:02:11.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:02:11.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:02:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:02:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:02:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:02:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:02:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-09-15T04:02:11.364 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:02:11.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T04:02:11.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=e0b11245-1e56-432a-8d23-204dc939ac75 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.90131/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:02:11.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-09-15T04:02:11.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T04:02:11.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:02:11.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:02:11.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:02:11.381 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:11.380+0000 7fb7e6b20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:11.382 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:11.382+0000 7fb7e6b20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:11.383 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:11.383+0000 7fb7e6b20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:11.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:02:11.762 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:02:11.906 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:11.906+0000 7fb7e6b20440 -1 Falling back to public interface 2024-09-15T04:02:12.125 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96503,v1:127.0.0.1:6827/96503] [v2:127.0.0.1:6828/96503,v1:127.0.0.1:6829/96503] exists,up 0e2cd43a-4a2b-4065-aa3c-7be7a727f47f 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:02:12.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:02:12.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:02:12.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:02:12.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:02:12.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:02:12.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:02:12.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:02:12.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:02:12.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:02:12.717 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:02:12.718 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:02:12.718 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T04:02:12.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:02:12.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:02:12.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:02:13.462 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:13.461+0000 7f3d29121440 -1 osd.0 25 log_to_monitors true 2024-09-15T04:02:14.620 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:14.620+0000 7f2ef7c12440 -1 osd.1 25 log_to_monitors true 2024-09-15T04:02:15.425 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:15.424+0000 7f6731f21440 -1 osd.2 24 log_to_monitors true 2024-09-15T04:02:16.111 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:16.110+0000 7fb7e6b20440 -1 osd.3 25 log_to_monitors true 2024-09-15T04:02:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084291 2024-09-15T04:02:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084291 2024-09-15T04:02:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-120259084291' 2024-09-15T04:02:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:02:17.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:02:18.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=133143986179 2024-09-15T04:02:18.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 133143986179 2024-09-15T04:02:18.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-120259084291 1-133143986179' 2024-09-15T04:02:18.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:02:18.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:02:18.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888067 2024-09-15T04:02:18.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888067 2024-09-15T04:02:18.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-146028888067' 2024-09-15T04:02:18.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:02:18.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:02:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=150323855363 2024-09-15T04:02:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 150323855363 2024-09-15T04:02:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-120259084291 1-133143986179 2-146028888067 3-150323855363' 2024-09-15T04:02:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:02:18.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-120259084291 2024-09-15T04:02:18.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:02:18.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:02:18.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-120259084291 2024-09-15T04:02:18.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:02:18.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084291 2024-09-15T04:02:18.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 120259084291' 2024-09-15T04:02:18.246 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 120259084291 2024-09-15T04:02:18.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:02:18.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084291 -lt 120259084291 2024-09-15T04:02:18.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:02:18.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-133143986179 2024-09-15T04:02:18.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:02:18.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:02:18.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-133143986179 2024-09-15T04:02:18.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:02:18.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=133143986179 2024-09-15T04:02:18.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 133143986179' 2024-09-15T04:02:18.598 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 133143986179 2024-09-15T04:02:18.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:02:18.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 133143986179 -lt 133143986179 2024-09-15T04:02:18.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:02:18.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888067 2024-09-15T04:02:18.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:02:18.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:02:18.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888067 2024-09-15T04:02:18.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:02:18.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888067 2024-09-15T04:02:18.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-09-15T04:02:18.965 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 146028888067 2024-09-15T04:02:18.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:02:19.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888067 -lt 146028888067 2024-09-15T04:02:19.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:02:19.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-150323855363 2024-09-15T04:02:19.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:02:19.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:02:19.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-150323855363 2024-09-15T04:02:19.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:02:19.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=150323855363 2024-09-15T04:02:19.313 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 150323855363 2024-09-15T04:02:19.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2024-09-15T04:02:19.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:02:19.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 150323855363 2024-09-15T04:02:19.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:02:20.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:02:20.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:02:21.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 150323855363 -lt 150323855363 2024-09-15T04:02:21.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:02:21.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:02:21.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:02:21.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T04:02:21.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:02:21.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:02:21.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:02:21.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:02:21.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:02:21.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:02:21.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:02:21.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:02:21.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:02:21.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:02:21.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:02:22.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:02:22.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:02:22.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:02:22.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:02:22.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:02:22.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:02:22.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:02:22.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:02:22.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:02:22.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:02:22.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:02:22.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:02:22.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:02:22.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:02:22.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-15T04:02:22.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:02:22.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:22.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:22.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90131 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/ec-error-rollforward 0 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:02:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:02:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:02:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:02:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:02:22.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:02:22.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:02:22.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:02:22.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:02:22.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:02:22.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:02:22.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:02:22.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:02:22.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:02:22.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:02:22.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-09-15T04:02:22.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:02:22.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:22.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.90131 2024-09-15T04:02:22.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.90131 2024-09-15T04:02:22.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:02:22.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:02:22.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:02:22.535 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:02:22.535 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:02:22.600 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-09-15T04:02:22.601 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-09-15T04:02:22.662 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:02:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-bench 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=6d9e0295-4e64-4eb8-ace4-2207d8afe826 --auth-supported=none ' 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-09-15T04:02:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-09-15T04:02:22.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:02:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:02:22.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:02:22.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:02:22.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:02:22.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:02:22.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:02:22.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:02:22.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:02:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:02:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:02:22.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:02:22.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:02:22.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:02:22.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-15T04:02:22.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:02:22.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:22.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:22.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.100077 2024-09-15T04:02:22.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:02:22.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:02:22.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-09-15T04:02:22.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:02:22.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:22.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:22.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.100077 2024-09-15T04:02:22.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:02:22.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:02:22.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-09-15T04:02:22.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-09-15T04:02:23.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:02:23.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:23.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:23.047 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:23.047 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:23.048 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:23.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.100077/$cluster-$name.asok' 2024-09-15T04:02:23.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100077/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:02:23.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:02:23.310 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:02:23.311 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:23.311 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:23.311 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.100077/ceph-mon.a.asok 2024-09-15T04:02:23.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:02:23.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.100077/ceph-mon.a.asok config get fsid 2024-09-15T04:02:23.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:02:23.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:02:23.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.100077/ceph-mon.a.asok 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:02:23.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.100077/ceph-mon.a.asok config get mon_host 2024-09-15T04:02:23.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-09-15T04:02:23.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-09-15T04:02:23.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:02:23.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:02:23.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:02:23.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-09-15T04:02:23.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:02:24.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.100077/$cluster-$name.asok' 2024-09-15T04:02:24.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:02:24.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.100077/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:02:24.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6d9e0295-4e64-4eb8-ace4-2207d8afe826 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:02:24.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.100077/$cluster-$name.asok' 2024-09-15T04:02:24.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.100077/$cluster-$name.asok' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:02:24.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=526df76b-4471-4084-8be4-dcc4fe8d7819 2024-09-15T04:02:24.331 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 526df76b-4471-4084-8be4-dcc4fe8d7819 2024-09-15T04:02:24.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 526df76b-4471-4084-8be4-dcc4fe8d7819' 2024-09-15T04:02:24.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:02:24.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQXOZmL0BLFBAAe9iNc+k4/ECtg26v9cCc7Q== 2024-09-15T04:02:24.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQXOZmL0BLFBAAe9iNc+k4/ECtg26v9cCc7Q=="}' 2024-09-15T04:02:24.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 526df76b-4471-4084-8be4-dcc4fe8d7819 -i td/osd-bench/0/new.json 2024-09-15T04:02:24.645 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:24.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-09-15T04:02:24.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6d9e0295-4e64-4eb8-ace4-2207d8afe826 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100077/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQXOZmL0BLFBAAe9iNc+k4/ECtg26v9cCc7Q== --osd-uuid 526df76b-4471-4084-8be4-dcc4fe8d7819 2024-09-15T04:02:24.678 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.677+0000 7fb467616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:24.683 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.682+0000 7fb467616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:24.686 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.683+0000 7fb467616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:24.686 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.684+0000 7fb467616440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-15T04:02:24.686 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.684+0000 7fb467616440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-15T04:02:24.686 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.684+0000 7fb467616440 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-09-15T04:02:24.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:24.723+0000 7fb467616440 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-09-15T04:02:29.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-09-15T04:02:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:02:29.761 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:02:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:02:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:02:30.281 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:02:30.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:02:30.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6d9e0295-4e64-4eb8-ace4-2207d8afe826 --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.100077/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:02:30.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:02:30.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:02:30.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:02:30.302 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:30.301+0000 7f2500f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:30.303+0000 7f2500f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:30.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:30.304+0000 7f2500f1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:30.667 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:02:30.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:02:30.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:30.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:30.821 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:30.821+0000 7f2500f1f440 -1 Falling back to public interface 2024-09-15T04:02:31.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:32.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:32.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:32.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:02:32.053 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:02:32.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:32.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:32.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:33.248 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:02:33.247+0000 7f2500f1f440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:02:33.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:33.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:33.454 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:02:33.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:02:33.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:33.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:33.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:34.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:34.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:34.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:02:34.829 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:02:34.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:34.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:35.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:36.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:36.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:36.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:02:36.314 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:02:36.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:36.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:36.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:37.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:37.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:37.668 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:02:37.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:02:37.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:37.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:38.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:39.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:39.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:39.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:02:39.077 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:02:39.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:39.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:39.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:40.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:40.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:40.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:02:40.438 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:02:40.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:40.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:40.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:41.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:41.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:41.962 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:02:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:02:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:42.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:43.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:43.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:43.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:02:43.326 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:02:43.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:43.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:43.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:44.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:44.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:44.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:02:44.726 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:02:44.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:44.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:45.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:46.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:46.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:46.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:02:46.518 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:02:46.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:46.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:47.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:48.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:48.363 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:02:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:02:48.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:48.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:49.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:50.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:50.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:50.208 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:02:50.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:02:50.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:50.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:51.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:52.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:02:52.158 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:02:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:53.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:54.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:54.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:54.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:02:54.247 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:02:54.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:54.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:55.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:56.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:56.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:56.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:02:56.186 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:02:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:56.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:02:57.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:02:57.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:02:57.562 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:02:57.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:02:57.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:02:57.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:02:58.221 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/101180,v1:127.0.0.1:6803/101180] [v2:127.0.0.1:6804/101180,v1:127.0.0.1:6805/101180] exists,up 526df76b-4471-4084-8be4-dcc4fe8d7819 2024-09-15T04:02:58.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:02:58.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:02:58.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:02:58.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-09-15T04:02:58.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-09-15T04:02:58.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-09-15T04:02:58.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-09-15T04:02:58.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-09-15T04:02:58.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-09-15T04:02:58.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-09-15T04:02:58.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-09-15T04:02:58.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-09-15T04:02:58.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-09-15T04:02:58.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=osd_bench_max_block_size 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T04:02:58.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-09-15T04:02:58.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T04:02:58.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T04:02:58.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-09-15T04:02:58.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=bench_small_size_max_iops 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T04:02:58.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-09-15T04:02:58.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T04:02:58.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T04:02:58.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: local dir=td/osd-bench 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1934: expect_failure: shift 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1935: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1936: expect_failure: shift 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1937: expect_failure: local success 2024-09-15T04:02:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1939: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-09-15T04:02:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1942: expect_failure: success=false 2024-09-15T04:02:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: false 2024-09-15T04:02:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-09-15T04:02:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: expect_failure: return 0 2024-09-15T04:02:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-09-15T04:03:49.053 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_written": 1073741824, 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout: "blocksize": 4194304, 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout: "elapsed_sec": 25.141178034999999, 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_per_sec": 42708492.915694036, 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout: "iops": 10.18249819652892 2024-09-15T04:03:49.054 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:03:49.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-09-15T04:03:49.513 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:03:49.513 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_written": 10, 2024-09-15T04:03:49.513 INFO:tasks.workunit.client.0.smithi138.stdout: "blocksize": 14456, 2024-09-15T04:03:49.514 INFO:tasks.workunit.client.0.smithi138.stdout: "elapsed_sec": 0.041959868999999997, 2024-09-15T04:03:49.514 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_per_sec": 238.32295567938976, 2024-09-15T04:03:49.514 INFO:tasks.workunit.client.0.smithi138.stdout: "iops": 0.016486092672896357 2024-09-15T04:03:49.514 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:03:49.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-09-15T04:03:49.755 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_written": 1, 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout: "blocksize": 14456, 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout: "elapsed_sec": 0.043489144, 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout: "bytes_per_sec": 22.994244264729605, 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout: "iops": 0.001590636708960266 2024-09-15T04:03:49.756 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:03:49.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:03:49.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:03:49.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:03:49.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:03:49.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:03:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:03:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:03:49.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:03:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:49.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:03:49.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:03:49.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:49.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:03:49.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:03:49.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-15T04:03:49.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:03:49.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:49.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:03:49.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.100077 2024-09-15T04:03:49.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:03:49.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:03:49.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:03:49.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-bench 0 2024-09-15T04:03:49.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:03:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:03:49.905 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:03:49.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:03:49.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:03:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:03:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:03:49.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:03:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:03:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:03:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:49.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:03:49.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:03:49.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-09-15T04:03:49.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:03:49.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:49.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.100077 2024-09-15T04:03:49.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.100077 2024-09-15T04:03:49.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:03:49.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:03:49.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:03:49.913 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:03:49.913 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:03:49.978 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-09-15T04:03:49.979 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-09-15T04:03:50.040 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:03:50.047 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:03:50.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:03:50.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:03:50.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-bluefs-volume-ops 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-09-15T04:03:50.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_bluestore2 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_bluestore_expand' 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:03:50.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:03:50.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:03:50.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:03:50.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:03:50.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:03:50.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:03:50.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:03:50.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:03:50.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:03:50.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:03:50.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:50.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:03:50.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:03:50.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:03:50.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:03:50.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:03:50.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:03:50.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:03:50.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:50.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:50.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:03:50.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:03:50.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:03:50.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T04:03:50.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:03:50.059 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:50.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:50.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102313 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-09-15T04:03:50.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-09-15T04:03:50.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-09-15T04:03:50.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:03:50.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:03:50.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T04:03:50.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:03:50.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:03:50.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:03:50.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:03:50.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:03:50.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get fsid 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:03:50.770 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:50.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:03:50.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:03:50.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get mon_host 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:03:50.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T04:03:50.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:03:51.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:03:51.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:03:51.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:03:51.678 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:03:51.678 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:51.679 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:51.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:03:51.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:03:51.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:03:51.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:03:51.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:03:51.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:03:51.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cf00fc34-feef-4921-b6dd-6f24bf03a3ef' 2024-09-15T04:03:51.709 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:03:51.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:03:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnXOZmEMjFKhAA4yVCXdjto4hXKflOVMvSlw== 2024-09-15T04:03:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnXOZmEMjFKhAA4yVCXdjto4hXKflOVMvSlw=="}' 2024-09-15T04:03:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cf00fc34-feef-4921-b6dd-6f24bf03a3ef -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:03:52.031 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:03:52.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:03:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnXOZmEMjFKhAA4yVCXdjto4hXKflOVMvSlw== --osd-uuid cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:03:52.065 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.064+0000 7fb506012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:03:52.076 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.076+0000 7fb506012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:03:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.077+0000 7fb506012440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:03:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.077+0000 7fb506012440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:03:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.077+0000 7fb506012440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:03:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.077+0000 7fb506012440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:03:52.117 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:03:52.116+0000 7fb506012440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T04:04:01.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T04:04:01.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:04:01.484 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:04:01.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:04:01.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:04:01.963 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:04:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:04:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:04:01.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:04:01.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:04:01.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:04:01.984 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:01.983+0000 7f97f2b5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:01.986 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:01.985+0000 7f97f2b5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:01.987 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:01.986+0000 7f97f2b5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:02.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:04:02.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:04:02.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:04:02.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:04:02.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:04:02.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:02.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:04:02.312 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:04:02.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:02.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:02.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:03.001 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:03.001+0000 7f97f2b5e440 -1 Falling back to public interface 2024-09-15T04:04:03.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:03.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:03.658 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:04:03.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:04:03.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:03.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:04.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:05.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:05.024 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:04:05.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:04:05.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:05.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:05.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:06.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:06.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:06.374 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:04:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:04:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:06.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:07.666 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:07.665+0000 7f97f2b5e440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:04:07.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:07.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:07.723 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:04:07.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:04:07.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:07.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:08.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:09.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:09.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:09.286 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:04:09.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:04:09.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:09.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:09.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:10.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:10.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:10.683 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:04:10.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:04:10.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:10.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:11.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:12.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:12.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:12.049 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:04:12.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:04:12.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:12.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:12.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:13.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:13.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:13.457 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:04:13.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:04:13.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:13.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:13.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:04:14.813 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:04:14.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:14.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:15.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:16.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:16.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:16.178 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:04:16.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:04:16.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:16.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:16.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:17.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:17.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:17.705 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:04:17.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:04:17.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:17.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:18.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:19.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:19.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:19.181 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:04:19.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:04:19.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:19.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:19.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:20.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:20.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:20.969 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:04:20.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:04:20.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:20.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:21.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:22.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:22.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:22.452 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:04:22.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:04:22.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:22.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:23.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:24.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:24.166 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:04:24.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:04:24.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:24.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:24.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:25.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:25.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:25.870 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:04:25.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:04:25.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:25.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:26.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:27.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:27.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:27.694 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:04:27.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:04:27.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:27.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:28.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:29.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:29.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:29.379 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:04:29.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:04:29.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:29.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:29.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:30.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:30.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:30.913 INFO:tasks.workunit.client.0.smithi138.stdout:19 2024-09-15T04:04:30.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T04:04:30.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:30.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:04:31.300 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103801,v1:127.0.0.1:6803/103801] [v2:127.0.0.1:6804/103801,v1:127.0.0.1:6805/103801] exists,up cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:04:31.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:04:31.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:04:31.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:04:31.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:04:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=103801 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T04:04:31.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:04:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:04:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:04:31.308 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:04:31.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3ca30242-fa4b-4024-8218-66d11439b6f1' 2024-09-15T04:04:31.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:04:31.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPXOZmcq7zEhAARSbbgyynXm7R7qPX3fKAwA== 2024-09-15T04:04:31.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPXOZmcq7zEhAARSbbgyynXm7R7qPX3fKAwA=="}' 2024-09-15T04:04:31.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3ca30242-fa4b-4024-8218-66d11439b6f1 -i td/osd-bluefs-volume-ops/1/new.json 2024-09-15T04:04:31.835 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:04:31.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-09-15T04:04:31.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPXOZmcq7zEhAARSbbgyynXm7R7qPX3fKAwA== --osd-uuid 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:04:31.870 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.869+0000 7f8a8aa67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:31.871 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.871+0000 7f8a8aa67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:31.872 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.872+0000 7f8a8aa67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:31.873 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.872+0000 7f8a8aa67440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-15T04:04:31.873 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.872+0000 7f8a8aa67440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-15T04:04:31.873 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.872+0000 7f8a8aa67440 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-09-15T04:04:31.925 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:31.924+0000 7f8a8aa67440 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-09-15T04:04:41.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-09-15T04:04:41.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:04:41.110 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:04:41.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:04:41.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:04:41.622 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:04:41.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:04:41.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:04:41.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:04:41.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:04:41.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:04:41.645 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:41.644+0000 7fc211630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:41.646 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:41.646+0000 7fc211630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:41.647 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:41.647+0000 7fc211630440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:04:42.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:42.011 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:04:42.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:04:42.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:42.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:42.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:42.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:42.677+0000 7fc211630440 -1 Falling back to public interface 2024-09-15T04:04:43.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:43.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:43.358 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:04:43.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:04:43.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:43.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:43.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:44.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:44.712 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:04:44.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:44.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:04:44.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:44.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:45.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:46.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:46.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:46.070 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:04:46.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:04:46.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:46.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:46.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:47.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:47.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:47.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:04:47.423 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:04:47.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:47.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:47.433 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:04:47.432+0000 7fc211630440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:04:47.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:48.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:48.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:48.886 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:04:48.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:04:48.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:48.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:49.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:50.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:50.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:50.328 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:04:50.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:04:50.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:50.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:50.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:51.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:51.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:51.734 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:04:51.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:04:51.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:51.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:52.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:53.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:53.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:53.193 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:04:53.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:04:53.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:53.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:53.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:54.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:54.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:54.642 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:04:54.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:04:54.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:54.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:55.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:56.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:56.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:56.033 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:04:56.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:04:56.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:56.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:56.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:57.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:57.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:57.508 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:04:57.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:04:57.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:57.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:57.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:04:58.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:04:58.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:04:58.874 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:04:58.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:04:58.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:04:58.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:04:59.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:00.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:00.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:00.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:05:00.465 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:05:00.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:00.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:01.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:02.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:02.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:02.193 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:05:02.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:05:02.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:02.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:02.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:03.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:03.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:03.862 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:05:03.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:05:03.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:03.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:04.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:05.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:05.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:05.911 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:05:05.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:05:05.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:05.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:06.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:07.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:07.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:07.570 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:05:07.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:05:07.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:07.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:08.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:09.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:09.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:09.503 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:05:09.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:05:09.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:09.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:09.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:10.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:10.921 INFO:tasks.workunit.client.0.smithi138.stdout:19 2024-09-15T04:05:10.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T04:05:10.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:10.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:05:11.335 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/106388,v1:127.0.0.1:6811/106388] [v2:127.0.0.1:6812/106388,v1:127.0.0.1:6813/106388] exists,up 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:05:11.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:05:11.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:05:11.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:05:11.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=106388 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:05:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:05:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:05:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:05:11.344 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:05:11.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d7d4b052-c5b5-4eaa-be71-cec8436c5154' 2024-09-15T04:05:11.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:05:11.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3XOZmujArFRAABVHQBw0C0MVlSWwdcuG2jQ== 2024-09-15T04:05:11.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3XOZmujArFRAABVHQBw0C0MVlSWwdcuG2jQ=="}' 2024-09-15T04:05:11.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d7d4b052-c5b5-4eaa-be71-cec8436c5154 -i td/osd-bluefs-volume-ops/2/new.json 2024-09-15T04:05:11.829 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:05:11.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-09-15T04:05:11.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3XOZmujArFRAABVHQBw0C0MVlSWwdcuG2jQ== --osd-uuid d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:05:11.864 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.862+0000 7f89c1d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:11.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.882+0000 7f89c1d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:11.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.883+0000 7f89c1d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:11.885 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.883+0000 7f89c1d21440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-15T04:05:11.885 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.883+0000 7f89c1d21440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-15T04:05:11.885 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.883+0000 7f89c1d21440 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-09-15T04:05:11.923 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:11.922+0000 7f89c1d21440 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-09-15T04:05:21.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-09-15T04:05:21.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:05:21.551 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:05:21.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:05:21.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:05:22.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:05:22.031 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:05:22.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:05:22.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:05:22.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:05:22.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:05:22.052 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:22.051+0000 7fd762f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:22.053 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:22.053+0000 7fd762f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:22.055 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:22.054+0000 7fd762f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:22.377 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:05:22.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:05:22.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:22.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:22.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:23.068 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:23.068+0000 7fd762f23440 -1 Falling back to public interface 2024-09-15T04:05:23.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:23.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:23.743 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:05:23.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:05:23.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:23.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:24.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:25.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:25.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:25.134 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:05:25.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:05:25.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:25.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:25.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:26.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:26.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:26.522 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:05:26.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:05:26.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:26.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:26.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:27.759 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:27.758+0000 7fd762f23440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:05:27.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:27.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:27.878 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:05:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:05:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:28.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:29.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:29.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:29.453 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:05:29.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:05:29.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:29.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:30.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:31.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:31.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:31.061 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:05:31.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:05:31.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:31.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:31.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:32.419 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:05:32.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:05:32.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:32.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:32.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:33.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:33.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:33.839 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:05:33.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:05:33.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:33.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:34.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:35.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:35.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:35.239 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:05:35.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:05:35.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:35.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:35.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:36.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:36.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:36.747 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:05:36.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:05:36.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:36.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:37.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:38.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:38.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:38.120 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:05:38.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:05:38.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:38.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:39.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:39.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:39.537 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:05:39.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:05:39.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:39.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:41.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:41.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:41.437 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:05:41.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:05:41.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:41.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:42.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:43.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:43.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:43.207 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:05:43.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:05:43.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:43.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:44.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:45.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:45.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:45.077 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:05:45.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:05:45.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:45.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:46.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:46.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:46.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:05:46.924 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:05:46.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:46.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:47.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:48.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:48.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:48.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:05:48.651 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:05:48.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:48.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:49.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:50.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:50.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:50.436 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:05:50.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:05:50.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:50.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:50.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:05:51.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:05:51.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:05:51.861 INFO:tasks.workunit.client.0.smithi138.stdout:19 2024-09-15T04:05:51.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T04:05:51.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:05:51.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:05:52.214 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/108975,v1:127.0.0.1:6819/108975] [v2:127.0.0.1:6820/108975,v1:127.0.0.1:6821/108975] exists,up d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:05:52.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:05:52.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:05:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:05:52.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T04:05:52.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=108975 2024-09-15T04:05:52.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-09-15T04:05:52.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:05:52.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:05:52.219 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:05:52.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:05:52.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:05:52.222 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:05:52.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b2c90f97-5756-450c-81d6-871a4aba7939' 2024-09-15T04:05:52.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:05:52.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgXeZmmkfVDRAAdNlekGGx15E+eqaKMf6W1g== 2024-09-15T04:05:52.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgXeZmmkfVDRAAdNlekGGx15E+eqaKMf6W1g=="}' 2024-09-15T04:05:52.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b2c90f97-5756-450c-81d6-871a4aba7939 -i td/osd-bluefs-volume-ops/3/new.json 2024-09-15T04:05:52.703 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:05:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-09-15T04:05:52.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgXeZmmkfVDRAAdNlekGGx15E+eqaKMf6W1g== --osd-uuid b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:05:52.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.737+0000 7f6ea1c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:52.739 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.739+0000 7f6ea1c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:52.741 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.740+0000 7f6ea1c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:05:52.741 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.740+0000 7f6ea1c0d440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-15T04:05:52.741 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.740+0000 7f6ea1c0d440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-15T04:05:52.741 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.740+0000 7f6ea1c0d440 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-09-15T04:05:52.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:05:52.818+0000 7f6ea1c0d440 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-09-15T04:06:02.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-09-15T04:06:02.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:06:02.154 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T04:06:02.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T04:06:02.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:06:02.642 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:06:02.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T04:06:02.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:06:02.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:06:02.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:06:02.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:06:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:06:02.663+0000 7f42b5433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:06:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:06:02.665+0000 7f42b5433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:06:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:06:02.666+0000 7f42b5433440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:06:02.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T04:06:02.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:06:02.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:06:02.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:02.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:03.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:03.707 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:06:03.707+0000 7f42b5433440 -1 Falling back to public interface 2024-09-15T04:06:04.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:04.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:04.339 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:06:04.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:06:04.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:04.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:04.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:05.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:05.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:05.691 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:06:05.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:06:05.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:05.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:06.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:07.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:07.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:07.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:06:07.087 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:06:07.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:07.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:07.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:08.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:06:08.463+0000 7f42b5433440 -1 osd.3 0 log_to_monitors true 2024-09-15T04:06:08.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:06:08.480 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:06:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:08.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:09.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:09.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:09.937 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:06:09.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:06:09.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:09.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:10.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:11.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:11.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:11.377 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:06:11.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:06:11.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:11.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:11.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:12.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:12.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:12.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:06:12.757 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:06:12.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:12.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:13.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:14.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:14.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:14.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:06:14.238 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:06:14.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:14.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:14.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:15.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:15.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:15.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:06:15.687 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:06:15.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:15.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:16.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:17.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:17.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:17.092 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:06:17.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:06:17.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:17.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:17.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:18.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:18.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:18.529 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:06:18.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:06:18.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:18.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:18.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:19.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:19.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:19.929 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:06:19.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:06:19.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:19.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:20.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:21.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:21.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:21.390 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:06:21.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:06:21.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:21.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:22.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:23.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:23.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:23.501 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:06:23.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:06:23.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:23.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:24.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:25.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:25.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:25.417 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:06:25.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:06:25.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:25.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:26.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:27.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:27.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:27.175 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:06:27.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:06:27.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:27.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:28.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:29.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:29.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:29.220 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:06:29.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:06:29.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:29.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:29.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:30.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:30.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:30.607 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:06:30.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:06:30.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:30.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:31.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:06:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:06:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:06:32.430 INFO:tasks.workunit.client.0.smithi138.stdout:19 2024-09-15T04:06:32.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-15T04:06:32.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:06:32.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:06:32.795 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/111563,v1:127.0.0.1:6827/111563] [v2:127.0.0.1:6828/111563,v1:127.0.0.1:6829/111563] exists,up b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:06:32.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:06:32.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:06:32.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:06:32.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T04:06:32.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=111563 2024-09-15T04:06:32.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-09-15T04:06:37.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-09-15T04:06:37.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-15T04:06:38.451 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T04:06:38.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:06:39.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T04:06:39.516 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:06:39.516 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T04:06:39.517 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_112941 2024-09-15T04:06:39.517 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:06:39.517 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:06:40.516 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 65 49 0.19138 0.191406 0.260501 0.287129 2024-09-15T04:06:41.516 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 125 109 0.212868 0.234375 0.263113 0.278503 2024-09-15T04:06:42.517 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 177 161 0.209615 0.203125 0.28607 0.282976 2024-09-15T04:06:43.517 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 231 215 0.20994 0.210938 0.323829 0.284522 2024-09-15T04:06:44.517 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 279 263 0.205446 0.1875 0.352202 0.292252 2024-09-15T04:06:45.517 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 337 321 0.208961 0.226562 0.307285 0.293831 2024-09-15T04:06:46.517 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 385 369 0.205891 0.1875 0.285559 0.296215 2024-09-15T04:06:47.517 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 439 423 0.206519 0.210938 0.268926 0.295847 2024-09-15T04:06:48.517 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 497 481 0.208745 0.226562 0.277592 0.294924 2024-09-15T04:06:49.517 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 551 535 0.208961 0.210938 0.309324 0.293745 2024-09-15T04:06:50.517 INFO:tasks.workunit.client.0.smithi138.stdout: 11 16 609 593 0.210559 0.226562 0.310996 0.294279 2024-09-15T04:06:51.518 INFO:tasks.workunit.client.0.smithi138.stdout: 12 16 657 641 0.208635 0.1875 0.337245 0.294974 2024-09-15T04:06:52.518 INFO:tasks.workunit.client.0.smithi138.stdout: 13 16 711 695 0.20881 0.210938 0.268161 0.294573 2024-09-15T04:06:53.518 INFO:tasks.workunit.client.0.smithi138.stdout: 14 16 769 753 0.210077 0.226562 0.298356 0.293965 2024-09-15T04:06:54.518 INFO:tasks.workunit.client.0.smithi138.stdout: 15 16 823 807 0.210132 0.210938 0.365418 0.293478 2024-09-15T04:06:55.518 INFO:tasks.workunit.client.0.smithi138.stdout: 16 16 881 865 0.211157 0.226562 0.246652 0.292752 2024-09-15T04:06:56.518 INFO:tasks.workunit.client.0.smithi138.stdout: 17 16 945 929 0.213439 0.25 0.280055 0.291436 2024-09-15T04:06:57.518 INFO:tasks.workunit.client.0.smithi138.stdout: 18 16 999 983 0.213299 0.210938 0.344192 0.290588 2024-09-15T04:06:58.518 INFO:tasks.workunit.client.0.smithi138.stdout: 19 16 1057 1041 0.213996 0.226562 0.26705 0.291299 2024-09-15T04:06:59.519 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:06:59.519291+0000 min lat: 0.187195 max lat: 0.40304 avg lat: 0.29126 2024-09-15T04:06:59.519 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:06:59.519 INFO:tasks.workunit.client.0.smithi138.stdout: 20 16 1105 1089 0.21267 0.1875 0.320369 0.29126 2024-09-15T04:07:00.519 INFO:tasks.workunit.client.0.smithi138.stdout: 21 16 1159 1143 0.212586 0.210938 0.277603 0.290769 2024-09-15T04:07:01.519 INFO:tasks.workunit.client.0.smithi138.stdout: 22 16 1223 1207 0.214285 0.25 0.258696 0.289492 2024-09-15T04:07:02.519 INFO:tasks.workunit.client.0.smithi138.stdout: 23 16 1281 1265 0.214817 0.226562 0.302045 0.289204 2024-09-15T04:07:03.521 INFO:tasks.workunit.client.0.smithi138.stdout: 24 16 1345 1329 0.216282 0.25 0.269014 0.288022 2024-09-15T04:07:04.519 INFO:tasks.workunit.client.0.smithi138.stdout: 25 16 1399 1383 0.216068 0.210938 0.27692 0.288276 2024-09-15T04:07:05.520 INFO:tasks.workunit.client.0.smithi138.stdout: 26 16 1457 1441 0.21647 0.226562 0.274781 0.287962 2024-09-15T04:07:06.519 INFO:tasks.workunit.client.0.smithi138.stdout: 27 16 1511 1495 0.216264 0.210938 0.251818 0.287415 2024-09-15T04:07:07.520 INFO:tasks.workunit.client.0.smithi138.stdout: 28 16 1575 1559 0.217468 0.25 0.265003 0.286576 2024-09-15T04:07:08.520 INFO:tasks.workunit.client.0.smithi138.stdout: 29 16 1623 1607 0.216434 0.1875 0.317332 0.286779 2024-09-15T04:07:09.520 INFO:tasks.workunit.client.0.smithi138.stdout: 30 16 1681 1665 0.216771 0.226562 0.29566 0.286648 2024-09-15T04:07:10.520 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 30.1264 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 1681 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.217962 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.019484 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.25 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.1875 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 55 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 4.98792 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 64 2024-09-15T04:07:10.521 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 48 2024-09-15T04:07:10.522 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.286321 2024-09-15T04:07:10.522 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0259395 2024-09-15T04:07:10.522 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.40304 2024-09-15T04:07:10.522 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.187195 2024-09-15T04:07:10.630 INFO:tasks.workunit.client.0.smithi138.stdout:after bench 2024-09-15T04:07:10.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-09-15T04:07:10.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 103801 2024-09-15T04:07:10.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-09-15T04:07:11.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 103801 2024-09-15T04:07:11.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (103801) - No such process 2024-09-15T04:07:11.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-09-15T04:07:12.195 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:07:12.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106388 2024-09-15T04:07:12.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-09-15T04:07:13.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 106388 2024-09-15T04:07:13.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (106388) - No such process 2024-09-15T04:07:13.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-09-15T04:07:13.695 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T04:07:13.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 108975 2024-09-15T04:07:13.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-09-15T04:07:14.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 108975 2024-09-15T04:07:14.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (108975) - No such process 2024-09-15T04:07:14.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-09-15T04:07:15.069 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T04:07:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111563 2024-09-15T04:07:15.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-09-15T04:07:16.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 111563 2024-09-15T04:07:16.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (111563) - No such process 2024-09-15T04:07:16.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-09-15T04:07:16.569 INFO:tasks.workunit.client.0.smithi138.stderr:osd.3 is already down. 2024-09-15T04:07:16.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:07:18.360 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:18.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:07:20.149 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:20.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:07:21.946 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:21.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:07:23.736 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:23.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-15T04:07:23.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-15T04:07:23.779 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:07:24.669 INFO:tasks.workunit.client.0.smithi138.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-15T04:07:24.669 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T04:07:24.669 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x100000000 : using 0x80533000(2.0 GiB) 2024-09-15T04:07:24.669 INFO:tasks.workunit.client.0.smithi138.stdout:Expanding DB/WAL... 2024-09-15T04:07:24.669 INFO:tasks.workunit.client.0.smithi138.stdout:0 : expanding to 0x536870912 2024-09-15T04:07:24.701 INFO:tasks.workunit.client.0.smithi138.stdout:0 : size label updated to 536870912 2024-09-15T04:07:24.701 INFO:tasks.workunit.client.0.smithi138.stdout:1 : expanding to 0x1073741824 2024-09-15T04:07:24.726 INFO:tasks.workunit.client.0.smithi138.stdout:1 : size label updated to 1073741824 2024-09-15T04:07:24.726 INFO:tasks.workunit.client.0.smithi138.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-09-15T04:07:24.910 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:24.909+0000 7fef09a0f780 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-15T04:07:24.910 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:24.909+0000 7fef09a0f780 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-09-15T04:07:29.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-09-15T04:07:29.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-09-15T04:07:30.015 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:07:30.917 INFO:tasks.workunit.client.0.smithi138.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-15T04:07:30.917 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T04:07:30.917 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x101000000 : using 0x8150d000(2.0 GiB) 2024-09-15T04:07:30.917 INFO:tasks.workunit.client.0.smithi138.stdout:Expanding DB/WAL... 2024-09-15T04:07:30.917 INFO:tasks.workunit.client.0.smithi138.stdout:0 : expanding to 0x536870912 2024-09-15T04:07:30.942 INFO:tasks.workunit.client.0.smithi138.stdout:0 : size label updated to 536870912 2024-09-15T04:07:30.942 INFO:tasks.workunit.client.0.smithi138.stdout:1 : expanding to 0x1073741824 2024-09-15T04:07:30.967 INFO:tasks.workunit.client.0.smithi138.stdout:1 : size label updated to 1073741824 2024-09-15T04:07:30.967 INFO:tasks.workunit.client.0.smithi138.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-09-15T04:07:31.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:31.175+0000 7f8fe5d1a780 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-15T04:07:31.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:31.175+0000 7f8fe5d1a780 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-09-15T04:07:36.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-09-15T04:07:36.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-09-15T04:07:36.262 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:07:37.164 INFO:tasks.workunit.client.0.smithi138.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-15T04:07:37.164 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T04:07:37.164 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x100020000 : using 0x8053a000(2.0 GiB) 2024-09-15T04:07:37.164 INFO:tasks.workunit.client.0.smithi138.stdout:Expanding DB/WAL... 2024-09-15T04:07:37.164 INFO:tasks.workunit.client.0.smithi138.stdout:0 : expanding to 0x536870912 2024-09-15T04:07:37.191 INFO:tasks.workunit.client.0.smithi138.stdout:0 : size label updated to 536870912 2024-09-15T04:07:37.191 INFO:tasks.workunit.client.0.smithi138.stdout:1 : expanding to 0x1073741824 2024-09-15T04:07:37.216 INFO:tasks.workunit.client.0.smithi138.stdout:1 : size label updated to 1073741824 2024-09-15T04:07:37.216 INFO:tasks.workunit.client.0.smithi138.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-09-15T04:07:37.451 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:37.450+0000 7f5087767780 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-15T04:07:37.451 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:37.450+0000 7f5087767780 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-09-15T04:07:42.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-09-15T04:07:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-09-15T04:07:42.728 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:07:43.638 INFO:tasks.workunit.client.0.smithi138.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-15T04:07:43.638 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-09-15T04:07:43.639 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0xfff00000 : using 0x8044a000(2.0 GiB) 2024-09-15T04:07:43.639 INFO:tasks.workunit.client.0.smithi138.stdout:Expanding DB/WAL... 2024-09-15T04:07:43.639 INFO:tasks.workunit.client.0.smithi138.stdout:0 : expanding to 0x536870912 2024-09-15T04:07:43.666 INFO:tasks.workunit.client.0.smithi138.stdout:0 : size label updated to 536870912 2024-09-15T04:07:43.666 INFO:tasks.workunit.client.0.smithi138.stdout:1 : expanding to 0x1073741824 2024-09-15T04:07:43.690 INFO:tasks.workunit.client.0.smithi138.stdout:1 : size label updated to 1073741824 2024-09-15T04:07:43.691 INFO:tasks.workunit.client.0.smithi138.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-09-15T04:07:43.984 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:43.984+0000 7f9604aca780 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-15T04:07:43.984 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:07:43.984+0000 7f9604aca780 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-09-15T04:07:49.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:07:50.845 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:50.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:07:52.633 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:52.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:07:54.424 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:54.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:07:56.212 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:07:56.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-15T04:07:56.237 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:07:58.004 INFO:tasks.workunit.client.0.smithi138.stdout:0 : device size 0x1ffff000 : using 0x11e00000(286 MiB) 2024-09-15T04:07:58.005 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0x1500000(21 MiB) 2024-09-15T04:07:58.005 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x100000000 : using 0x533000(5.2 MiB) 2024-09-15T04:07:58.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-09-15T04:07:58.022 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:01.787 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-15T04:08:01.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:08:03.101 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:03.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-15T04:08:03.128 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:05.141 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T04:08:05.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:08:06.430 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:06.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-09-15T04:08:06.454 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:10.471 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-15T04:08:10.472 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-15T04:08:10.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:08:11.280 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:11.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-09-15T04:08:11.305 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:11.305 INFO:tasks.workunit.client.0.smithi138.stderr:Migrate to WAL device isn't supported. 2024-09-15T04:08:11.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-15T04:08:11.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:08:13.086 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:13.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-15T04:08:13.112 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:16.877 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-15T04:08:16.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:08:18.182 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:18.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-09-15T04:08:18.206 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:08:21.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:08:23.028 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:08:23.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:08:23.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:08:23.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:08:23.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:08:23.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:08:23.038 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:08:23.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:08:23.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:08:23.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:08:23.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:08:23.040 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:08:23.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:08:23.061 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:23.061+0000 7f1fe6b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:23.063 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:23.063+0000 7f1fe6b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:23.065 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:23.065+0000 7f1fe6b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:08:23.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:23.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:08:23.396 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:08:23.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:23.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:08:23.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:23.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:23.829+0000 7f1fe6b1d440 -1 Falling back to public interface 2024-09-15T04:08:24.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:24.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:24.746 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:08:24.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:08:24.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:24.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:08:25.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:26.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:26.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:26.099 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:08:26.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:08:26.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:26.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:08:26.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:27.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:27.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:27.442 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:08:27.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:08:27.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:27.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:08:27.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:27.925 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:27.924+0000 7f1fe6b1d440 -1 osd.0 24 log_to_monitors true 2024-09-15T04:08:28.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:28.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:28.794 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:08:28.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:08:28.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:28.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:08:29.137 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 34 up_thru 34 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/125206,v1:127.0.0.1:6803/125206] [v2:127.0.0.1:6804/125206,v1:127.0.0.1:6805/125206] exists,up cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:08:29.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:08:29.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:08:29.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:08:29.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=125206 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T04:08:29.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:08:29.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:08:29.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T04:08:29.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T04:08:29.146 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:08:29.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:08:29.147 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T04:08:29.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T04:08:29.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:08:29.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:08:29.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:08:29.163 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:29.162+0000 7f9145465440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:29.190 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:29.189+0000 7f9145465440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:29.191+0000 7f9145465440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:29.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:08:29.537 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:08:29.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:29.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:08:29.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:29.956 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:29.955+0000 7f9145465440 -1 Falling back to public interface 2024-09-15T04:08:30.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:30.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:30.892 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:08:30.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:08:30.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:30.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:08:31.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:32.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:08:32.240 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:08:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:08:32.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:33.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:08:33.589 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:08:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:08:33.860 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:33.859+0000 7f9145465440 -1 osd.1 27 log_to_monitors true 2024-09-15T04:08:34.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:35.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:35.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:35.061 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:08:35.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:08:35.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:35.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:08:35.425 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 37 up_thru 37 down_at 28 last_clean_interval [10,27) [v2:127.0.0.1:6810/126187,v1:127.0.0.1:6811/126187] [v2:127.0.0.1:6812/126187,v1:127.0.0.1:6813/126187] exists,up 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:08:35.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:08:35.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:08:35.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:08:35.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=126187 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:08:35.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:08:35.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:08:35.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T04:08:35.432 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:08:35.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T04:08:35.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:08:35.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T04:08:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T04:08:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:08:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:08:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:08:35.450 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:35.449+0000 7f405b011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:35.452 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:35.452+0000 7f405b011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:35.453 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:35.453+0000 7f405b011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:08:35.777 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:08:35.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:35.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:08:35.969 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:35.969+0000 7f405b011440 -1 Falling back to public interface 2024-09-15T04:08:36.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:37.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:37.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:37.128 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:08:37.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:08:37.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:37.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:08:37.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:38.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:38.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:38.499 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:08:38.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:08:38.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:38.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:08:38.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:38.882 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:38.881+0000 7f405b011440 -1 osd.2 29 log_to_monitors true 2024-09-15T04:08:39.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:39.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:39.856 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:08:39.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:08:39.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:39.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:08:40.299 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 40 up_thru 40 down_at 30 last_clean_interval [15,29) [v2:127.0.0.1:6818/127168,v1:127.0.0.1:6819/127168] [v2:127.0.0.1:6820/127168,v1:127.0.0.1:6821/127168] exists,up d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=127168 2024-09-15T04:08:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:08:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:08:40.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:08:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T04:08:40.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T04:08:40.312 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:08:40.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:08:40.325 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:40.325+0000 7f29fee0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:40.327 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:40.327+0000 7f29fee0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:40.329 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:40.328+0000 7f29fee0f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:40.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:08:40.700 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:08:40.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:40.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:08:41.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:41.092+0000 7f29fee0f440 -1 Falling back to public interface 2024-09-15T04:08:42.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:42.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:42.054 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:08:42.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:08:42.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:42.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:08:42.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:43.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:43.413 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:08:43.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:43.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:08:43.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:43.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:08:43.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:44.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:44.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:44.764 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:08:44.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:08:44.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:44.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:08:44.793 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:08:44.792+0000 7f29fee0f440 -1 osd.3 31 log_to_monitors true 2024-09-15T04:08:45.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:08:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:08:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:08:46.280 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:08:46.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:08:46.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:08:46.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:08:46.840 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 44 up_thru 44 down_at 32 last_clean_interval [20,31) [v2:127.0.0.1:6826/128114,v1:127.0.0.1:6827/128114] [v2:127.0.0.1:6828/128114,v1:127.0.0.1:6829/128114] exists,up b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:08:46.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:08:46.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:08:46.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:08:46.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T04:08:46.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=128114 2024-09-15T04:08:46.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-09-15T04:08:46.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:08:46.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:08:46.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:08:46.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:08:46.842 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:08:46.843 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:08:46.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:08:46.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:08:46.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:08:46.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:08:47.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:08:47.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:08:47.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888070 2024-09-15T04:08:47.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888070 2024-09-15T04:08:47.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888070' 2024-09-15T04:08:47.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:08:47.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:08:47.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=158913789957 2024-09-15T04:08:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 158913789957 2024-09-15T04:08:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888070 1-158913789957' 2024-09-15T04:08:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:08:47.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:08:47.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=171798691844 2024-09-15T04:08:47.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 171798691844 2024-09-15T04:08:47.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888070 1-158913789957 2-171798691844' 2024-09-15T04:08:47.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:08:47.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:08:47.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-15T04:08:47.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-15T04:08:47.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-146028888070 1-158913789957 2-171798691844 3-188978561027' 2024-09-15T04:08:47.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:08:47.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-146028888070 2024-09-15T04:08:47.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:08:47.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:08:47.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-146028888070 2024-09-15T04:08:47.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:08:47.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888070 2024-09-15T04:08:47.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 146028888070' 2024-09-15T04:08:47.796 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 146028888070 2024-09-15T04:08:47.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:08:48.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888068 -lt 146028888070 2024-09-15T04:08:48.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:08:49.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:08:49.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:08:49.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888071 -lt 146028888070 2024-09-15T04:08:49.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:08:49.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-158913789957 2024-09-15T04:08:49.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:08:49.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:08:49.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-158913789957 2024-09-15T04:08:49.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:08:49.513 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 158913789957 2024-09-15T04:08:49.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=158913789957 2024-09-15T04:08:49.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 158913789957' 2024-09-15T04:08:49.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:08:49.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 158913789957 -lt 158913789957 2024-09-15T04:08:49.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:08:49.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-171798691844 2024-09-15T04:08:49.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:08:49.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:08:49.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-171798691844 2024-09-15T04:08:49.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:08:49.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=171798691844 2024-09-15T04:08:49.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 171798691844' 2024-09-15T04:08:49.883 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 171798691844 2024-09-15T04:08:49.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:08:50.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 171798691845 -lt 171798691844 2024-09-15T04:08:50.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:08:50.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-188978561027 2024-09-15T04:08:50.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:08:50.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:08:50.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-188978561027 2024-09-15T04:08:50.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:08:50.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-15T04:08:50.234 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 188978561027 2024-09-15T04:08:50.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 188978561027' 2024-09-15T04:08:50.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:08:50.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-15T04:08:50.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:08:50.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:08:50.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:08:51.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T04:08:51.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:08:51.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:08:51.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:08:51.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:08:51.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:08:51.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:08:51.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:08:51.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T04:08:51.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:08:51.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:08:51.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:08:51.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T04:08:51.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:08:51.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:08:51.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T04:08:51.907 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:08:51.907 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T04:08:51.907 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_129463 2024-09-15T04:08:51.907 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:08:51.907 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:08:52.907 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 88 72 0.281211 0.28125 0.183337 0.20152 2024-09-15T04:08:53.907 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 167 151 0.294886 0.308594 0.148413 0.201997 2024-09-15T04:08:54.907 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 241 225 0.292935 0.289062 0.19455 0.20758 2024-09-15T04:08:55.907 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 321 305 0.297817 0.3125 0.209357 0.206608 2024-09-15T04:08:56.908 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 388 372 0.290594 0.261719 0.234658 0.210691 2024-09-15T04:08:57.908 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 450 434 0.28252 0.242188 0.220607 0.211865 2024-09-15T04:08:58.908 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 517 501 0.279545 0.261719 0.250328 0.219614 2024-09-15T04:08:59.908 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 573 557 0.271944 0.21875 0.219328 0.224688 2024-09-15T04:09:00.908 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 634 618 0.268201 0.238281 0.319432 0.230073 2024-09-15T04:09:01.908 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 689 673 0.262862 0.214844 0.21674 0.233864 2024-09-15T04:09:02.908 INFO:tasks.workunit.client.0.smithi138.stdout: 11 16 744 728 0.258494 0.214844 0.142553 0.236758 2024-09-15T04:09:03.909 INFO:tasks.workunit.client.0.smithi138.stdout: 12 16 796 780 0.253878 0.203125 0.304501 0.242805 2024-09-15T04:09:04.909 INFO:tasks.workunit.client.0.smithi138.stdout: 13 16 851 835 0.250873 0.214844 0.178372 0.246622 2024-09-15T04:09:05.909 INFO:tasks.workunit.client.0.smithi138.stdout: 14 16 907 891 0.248576 0.21875 0.224877 0.249126 2024-09-15T04:09:06.909 INFO:tasks.workunit.client.0.smithi138.stdout: 15 16 967 951 0.247628 0.234375 0.159232 0.250822 2024-09-15T04:09:07.909 INFO:tasks.workunit.client.0.smithi138.stdout: 16 16 1039 1023 0.249727 0.28125 0.193376 0.249533 2024-09-15T04:09:08.909 INFO:tasks.workunit.client.0.smithi138.stdout: 17 16 1095 1079 0.247904 0.21875 0.249617 0.249289 2024-09-15T04:09:09.909 INFO:tasks.workunit.client.0.smithi138.stdout: 18 16 1174 1158 0.251274 0.308594 0.192545 0.246369 2024-09-15T04:09:10.909 INFO:tasks.workunit.client.0.smithi138.stdout: 19 16 1251 1235 0.253878 0.300781 0.217895 0.243969 2024-09-15T04:09:11.909 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:09:11.910038+0000 min lat: 0.0962314 max lat: 0.443383 avg lat: 0.241601 2024-09-15T04:09:11.910 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:09:11.910 INFO:tasks.workunit.client.0.smithi138.stdout: 20 16 1332 1316 0.257002 0.316406 0.196286 0.241601 2024-09-15T04:09:12.910 INFO:tasks.workunit.client.0.smithi138.stdout: 21 16 1399 1383 0.257224 0.261719 0.242784 0.241517 2024-09-15T04:09:13.910 INFO:tasks.workunit.client.0.smithi138.stdout: 22 16 1455 1439 0.255474 0.21875 0.331623 0.240935 2024-09-15T04:09:14.910 INFO:tasks.workunit.client.0.smithi138.stdout: 23 16 1515 1499 0.254555 0.234375 0.176937 0.243457 2024-09-15T04:09:15.910 INFO:tasks.workunit.client.0.smithi138.stdout: 24 16 1599 1583 0.257619 0.328125 0.25224 0.241853 2024-09-15T04:09:16.910 INFO:tasks.workunit.client.0.smithi138.stdout: 25 16 1669 1653 0.258251 0.273438 0.209032 0.240933 2024-09-15T04:09:17.910 INFO:tasks.workunit.client.0.smithi138.stdout: 26 16 1759 1743 0.261838 0.351562 0.176238 0.238282 2024-09-15T04:09:18.910 INFO:tasks.workunit.client.0.smithi138.stdout: 27 16 1827 1811 0.261977 0.265625 0.267994 0.237697 2024-09-15T04:09:19.910 INFO:tasks.workunit.client.0.smithi138.stdout: 28 16 1900 1884 0.262804 0.285156 0.229534 0.237046 2024-09-15T04:09:20.911 INFO:tasks.workunit.client.0.smithi138.stdout: 29 16 1963 1947 0.262227 0.246094 0.136341 0.23705 2024-09-15T04:09:21.913 INFO:tasks.workunit.client.0.smithi138.stdout: 30 16 2045 2029 0.264162 0.320312 0.140892 0.235909 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 30.2025 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 2045 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.26449 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.0413206 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.351562 2024-09-15T04:09:22.911 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.203125 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 67 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 10.5781 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 90 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 52 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.235963 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0605051 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.595052 2024-09-15T04:09:22.912 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0962314 2024-09-15T04:09:22.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125206 2024-09-15T04:09:22.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-09-15T04:09:23.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 125206 2024-09-15T04:09:23.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (125206) - No such process 2024-09-15T04:09:23.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-09-15T04:09:24.426 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:09:24.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126187 2024-09-15T04:09:24.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-09-15T04:09:25.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 126187 2024-09-15T04:09:25.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (126187) - No such process 2024-09-15T04:09:25.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-09-15T04:09:25.793 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T04:09:25.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127168 2024-09-15T04:09:25.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-09-15T04:09:26.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 127168 2024-09-15T04:09:26.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (127168) - No such process 2024-09-15T04:09:26.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-09-15T04:09:27.194 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T04:09:27.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128114 2024-09-15T04:09:27.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-09-15T04:09:28.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 128114 2024-09-15T04:09:28.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (128114) - No such process 2024-09-15T04:09:28.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-09-15T04:09:28.691 INFO:tasks.workunit.client.0.smithi138.stderr:osd.3 is already down. 2024-09-15T04:09:28.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:09:29.981 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-09-15T04:09:30.246 INFO:tasks.workunit.client.0.smithi138.stderr:512+0 records in 2024-09-15T04:09:30.246 INFO:tasks.workunit.client.0.smithi138.stderr:512+0 records out 2024-09-15T04:09:30.246 INFO:tasks.workunit.client.0.smithi138.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.253521 s, 2.1 GB/s 2024-09-15T04:09:30.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-09-15T04:09:30.262 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:35.026 INFO:tasks.workunit.client.0.smithi138.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-09-15T04:09:35.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:09:36.811 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:36.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:09:38.111 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:38.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-09-15T04:09:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:09:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:09:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.488184 s, 2.2 GB/s 2024-09-15T04:09:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-09-15T04:09:38.627 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:47.143 INFO:tasks.workunit.client.0.smithi138.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-09-15T04:09:47.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-09-15T04:09:47.163 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:48.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:09:50.712 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:50.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:09:51.507 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:51.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-09-15T04:09:51.528 INFO:tasks.workunit.client.0.smithi138.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T04:09:51.528 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:52.541 INFO:tasks.workunit.client.0.smithi138.stdout:DB device added 2024-09-15T04:09:52.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-09-15T04:09:52.560 INFO:tasks.workunit.client.0.smithi138.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T04:09:52.561 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:53.823 INFO:tasks.workunit.client.0.smithi138.stdout:WAL device added 2024-09-15T04:09:53.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-09-15T04:09:53.844 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:09:58.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:09:59.930 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:09:59.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:10:01.225 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:10:01.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-09-15T04:10:01.249 INFO:tasks.workunit.client.0.smithi138.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-09-15T04:10:01.249 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:10:02.761 INFO:tasks.workunit.client.0.smithi138.stdout:WAL device added 2024-09-15T04:10:02.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-09-15T04:10:02.779 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:10:06.551 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-09-15T04:10:06.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:10:07.854 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:10:07.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:10:07.865 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:10:07.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:10:07.867 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:10:07.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:10:07.887 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:07.887+0000 7f33aaa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:07.908 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:07.907+0000 7f33aaa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:07.909 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:07.909+0000 7f33aaa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:10:08.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:08.218 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:10:08.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:10:08.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:08.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:10:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:08.924+0000 7f33aaa08440 -1 Falling back to public interface 2024-09-15T04:10:09.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:09.571 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:10:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:10:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:09.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:10:09.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:10.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:10.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:10.951 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:10:10.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:10:10.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:10.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:10:11.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:12.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:12.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:12.295 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:10:12.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:10:12.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:12.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:10:12.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:13.576 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:13.576+0000 7f33aaa08440 -1 osd.0 46 log_to_monitors true 2024-09-15T04:10:13.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:13.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:13.646 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:10:13.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:10:13.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:13.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:10:14.077 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 56 up_thru 44 down_at 47 last_clean_interval [34,46) [v2:127.0.0.1:6802/135648,v1:127.0.0.1:6803/135648] [v2:127.0.0.1:6804/135648,v1:127.0.0.1:6805/135648] exists,up cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:10:14.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:10:14.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:10:14.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:10:14.078 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:10:14.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=135648 2024-09-15T04:10:14.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T04:10:14.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:10:14.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:10:14.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:10:14.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T04:10:14.084 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:10:14.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:10:14.102 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:14.102+0000 7ff390a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:14.185 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:14.184+0000 7ff390a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:14.185 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:14.185+0000 7ff390a1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:10:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:10:14.462 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:10:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:14.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:15.199 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:15.199+0000 7ff390a1d440 -1 Falling back to public interface 2024-09-15T04:10:15.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:15.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:15.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:10:15.839 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:10:15.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:15.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:16.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:17.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:17.188 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:10:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:10:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:17.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:17.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:18.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:18.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:18.568 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:10:18.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:10:18.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:18.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:18.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:19.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:19.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:19.915 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:10:19.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:10:19.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:19.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:20.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:20.376 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:20.375+0000 7ff390a1d440 -1 osd.1 49 log_to_monitors true 2024-09-15T04:10:21.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:21.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:21.272 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:10:21.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:10:21.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:21.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:10:21.827 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 59 up_thru 59 down_at 50 last_clean_interval [37,49) [v2:127.0.0.1:6810/136656,v1:127.0.0.1:6811/136656] [v2:127.0.0.1:6812/136656,v1:127.0.0.1:6813/136656] exists,up 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:10:21.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:10:21.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:10:21.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:10:21.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T04:10:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=136656 2024-09-15T04:10:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T04:10:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:10:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:10:21.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:21.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T04:10:21.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T04:10:21.837 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:10:21.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:10:21.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T04:10:21.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T04:10:21.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:10:21.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:10:21.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:10:21.852 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:21.851+0000 7fd2b720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:21.862 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:21.862+0000 7fd2b720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:21.863 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:21.863+0000 7fd2b720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:22.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T04:10:22.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:10:22.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:10:22.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:10:22.183 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:10:22.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:10:22.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:22.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:10:22.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:22.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:22.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:22.877 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:22.877+0000 7fd2b720a440 -1 Falling back to public interface 2024-09-15T04:10:23.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:23.529 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:10:23.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:23.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:10:23.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:23.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:23.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:24.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:24.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:24.936 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:10:24.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:10:24.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:24.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:25.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:26.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:26.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:26.293 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:10:26.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:10:26.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:26.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:26.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:27.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:27.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:27.649 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:10:27.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:10:27.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:27.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:28.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:28.157 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:28.157+0000 7fd2b720a440 -1 osd.2 51 log_to_monitors true 2024-09-15T04:10:29.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:29.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:29.014 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:10:29.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:10:29.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:29.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:10:29.757 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 62 up_thru 62 down_at 52 last_clean_interval [40,51) [v2:127.0.0.1:6818/137659,v1:127.0.0.1:6819/137659] [v2:127.0.0.1:6820/137659,v1:127.0.0.1:6821/137659] exists,up d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:10:29.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:10:29.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:10:29.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:10:29.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=137659 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T04:10:29.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:10:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:10:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T04:10:29.764 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:10:29.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:10:29.782 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:29.782+0000 7f8f6485a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:29.784 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:29.784+0000 7f8f6485a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:29.785 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:29.785+0000 7f8f6485a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:10:30.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:10:30.121 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:10:30.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:30.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:10:30.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:30.550 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:30.550+0000 7f8f6485a440 -1 Falling back to public interface 2024-09-15T04:10:31.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:31.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:31.530 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:10:31.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:10:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:10:31.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:32.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:32.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:32.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:10:32.924 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:10:32.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:32.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:10:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:34.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:34.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:34.269 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:10:34.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:10:34.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:34.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:10:34.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:10:34.750 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:10:34.749+0000 7f8f6485a440 -1 osd.3 53 log_to_monitors true 2024-09-15T04:10:35.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:10:35.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:10:35.627 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:10:35.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:10:35.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:10:35.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:10:36.139 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 66 up_thru 66 down_at 54 last_clean_interval [44,53) [v2:127.0.0.1:6826/138675,v1:127.0.0.1:6827/138675] [v2:127.0.0.1:6828/138675,v1:127.0.0.1:6829/138675] exists,up b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:10:36.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:10:36.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:10:36.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:10:36.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T04:10:36.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=138675 2024-09-15T04:10:36.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T04:10:36.166 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:10:36.166 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T04:10:36.166 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_139641 2024-09-15T04:10:36.166 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:10:36.166 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:10:37.166 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 52 36 0.140614 0.140625 0.317608 0.346776 2024-09-15T04:10:38.166 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 105 89 0.173811 0.207031 0.389347 0.334199 2024-09-15T04:10:39.166 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 161 145 0.18878 0.21875 0.280982 0.310006 2024-09-15T04:10:40.166 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 216 200 0.19529 0.214844 0.275745 0.309089 2024-09-15T04:10:41.166 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 275 259 0.20232 0.230469 0.193921 0.303545 2024-09-15T04:10:42.167 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 324 308 0.200497 0.191406 0.3187 0.302441 2024-09-15T04:10:43.167 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 373 357 0.199194 0.191406 0.30859 0.306335 2024-09-15T04:10:44.167 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 424 408 0.199194 0.199219 0.352426 0.305084 2024-09-15T04:10:45.167 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 484 468 0.203099 0.234375 0.155956 0.304267 2024-09-15T04:10:46.167 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 534 518 0.202318 0.195312 0.400345 0.3036 2024-09-15T04:10:47.167 INFO:tasks.workunit.client.0.smithi138.stdout: 11 16 592 576 0.20452 0.226562 0.260846 0.301508 2024-09-15T04:10:48.167 INFO:tasks.workunit.client.0.smithi138.stdout: 12 16 628 612 0.199193 0.140625 0.315503 0.308565 2024-09-15T04:10:49.168 INFO:tasks.workunit.client.0.smithi138.stdout: 13 16 689 673 0.202198 0.238281 0.328464 0.306163 2024-09-15T04:10:50.168 INFO:tasks.workunit.client.0.smithi138.stdout: 14 16 745 729 0.203379 0.21875 0.293776 0.304927 2024-09-15T04:10:51.168 INFO:tasks.workunit.client.0.smithi138.stdout: 15 16 800 784 0.204141 0.214844 0.243569 0.303558 2024-09-15T04:10:52.168 INFO:tasks.workunit.client.0.smithi138.stdout: 16 16 851 835 0.203832 0.199219 0.207304 0.303607 2024-09-15T04:10:53.168 INFO:tasks.workunit.client.0.smithi138.stdout: 17 16 899 883 0.20287 0.1875 0.371161 0.303742 2024-09-15T04:10:54.168 INFO:tasks.workunit.client.0.smithi138.stdout: 18 16 953 937 0.203317 0.210938 0.301911 0.304335 2024-09-15T04:10:55.168 INFO:tasks.workunit.client.0.smithi138.stdout: 19 16 1014 998 0.205155 0.238281 0.300326 0.302972 2024-09-15T04:10:56.168 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:10:56.169058+0000 min lat: 0.155956 max lat: 0.578065 avg lat: 0.302383 2024-09-15T04:10:56.169 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:10:56.169 INFO:tasks.workunit.client.0.smithi138.stdout: 20 16 1065 1049 0.204858 0.199219 0.222923 0.302383 2024-09-15T04:10:57.169 INFO:tasks.workunit.client.0.smithi138.stdout: 21 16 1122 1106 0.205704 0.222656 0.312295 0.302245 2024-09-15T04:10:58.169 INFO:tasks.workunit.client.0.smithi138.stdout: 22 16 1179 1163 0.206473 0.222656 0.161441 0.301198 2024-09-15T04:10:59.169 INFO:tasks.workunit.client.0.smithi138.stdout: 23 16 1237 1221 0.207346 0.226562 0.198518 0.300291 2024-09-15T04:11:00.169 INFO:tasks.workunit.client.0.smithi138.stdout: 24 16 1288 1272 0.207006 0.199219 0.184712 0.299799 2024-09-15T04:11:01.169 INFO:tasks.workunit.client.0.smithi138.stdout: 25 16 1344 1328 0.207475 0.21875 0.324636 0.299253 2024-09-15T04:11:02.169 INFO:tasks.workunit.client.0.smithi138.stdout: 26 16 1404 1388 0.208509 0.234375 0.285685 0.298758 2024-09-15T04:11:03.169 INFO:tasks.workunit.client.0.smithi138.stdout: 27 16 1456 1440 0.208308 0.203125 0.174291 0.297671 2024-09-15T04:11:04.169 INFO:tasks.workunit.client.0.smithi138.stdout: 28 16 1505 1489 0.207704 0.191406 0.293031 0.299287 2024-09-15T04:11:05.169 INFO:tasks.workunit.client.0.smithi138.stdout: 29 16 1561 1545 0.208084 0.21875 0.333577 0.298957 2024-09-15T04:11:06.169 INFO:tasks.workunit.client.0.smithi138.stdout: 30 16 1614 1598 0.208049 0.207031 0.257833 0.298661 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 30.1694 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 1614 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.208976 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.0236594 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.238281 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.140625 2024-09-15T04:11:07.170 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 53 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 6.05682 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 61 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 36 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.298624 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0585133 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.578065 2024-09-15T04:11:07.171 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.151532 2024-09-15T04:11:07.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135648 2024-09-15T04:11:07.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-09-15T04:11:08.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 135648 2024-09-15T04:11:08.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (135648) - No such process 2024-09-15T04:11:08.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-09-15T04:11:08.675 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:11:08.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136656 2024-09-15T04:11:08.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-09-15T04:11:09.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 136656 2024-09-15T04:11:09.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (136656) - No such process 2024-09-15T04:11:09.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-09-15T04:11:10.084 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T04:11:10.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137659 2024-09-15T04:11:10.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-09-15T04:11:11.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 137659 2024-09-15T04:11:11.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (137659) - No such process 2024-09-15T04:11:11.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-09-15T04:11:11.453 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T04:11:11.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138675 2024-09-15T04:11:11.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-09-15T04:11:12.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 138675 2024-09-15T04:11:12.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (138675) - No such process 2024-09-15T04:11:12.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-09-15T04:11:12.953 INFO:tasks.workunit.client.0.smithi138.stderr:osd.3 is already down. 2024-09-15T04:11:12.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:11:14.742 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:14.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-09-15T04:11:15.244 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:11:15.245 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:11:15.245 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.490358 s, 2.2 GB/s 2024-09-15T04:11:15.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-09-15T04:11:15.260 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:11:24.526 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-15T04:11:24.526 INFO:tasks.workunit.client.0.smithi138.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-09-15T04:11:24.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:11:26.314 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-09-15T04:11:26.582 INFO:tasks.workunit.client.0.smithi138.stderr:512+0 records in 2024-09-15T04:11:26.582 INFO:tasks.workunit.client.0.smithi138.stderr:512+0 records out 2024-09-15T04:11:26.582 INFO:tasks.workunit.client.0.smithi138.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.255511 s, 2.1 GB/s 2024-09-15T04:11:26.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-09-15T04:11:26.598 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:11:34.113 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-09-15T04:11:34.114 INFO:tasks.workunit.client.0.smithi138.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-09-15T04:11:34.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-09-15T04:11:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:11:35.982 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:35.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:11:37.760 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:37.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-09-15T04:11:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:11:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:11:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.495178 s, 2.2 GB/s 2024-09-15T04:11:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-09-15T04:11:38.280 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:11:49.549 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T04:11:49.549 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-09-15T04:11:49.549 INFO:tasks.workunit.client.0.smithi138.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-09-15T04:11:49.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-09-15T04:11:49.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:11:50.961 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:50.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-09-15T04:11:50.986 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:11:54.500 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-09-15T04:11:54.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-09-15T04:11:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-09-15T04:11:55.405 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:55.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-09-15T04:11:57.195 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:57.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-15T04:11:57.689 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:11:57.690 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:11:57.690 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.484645 s, 2.2 GB/s 2024-09-15T04:11:57.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-15T04:11:57.705 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:11:57.705 INFO:tasks.workunit.client.0.smithi138.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-09-15T04:11:57.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-09-15T04:11:57.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:11:59.484 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:11:59.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-09-15T04:12:06.945 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:12:06.946 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:12:06.946 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.33661 s, 146 MB/s 2024-09-15T04:12:06.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-09-15T04:12:06.964 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:12:11.481 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-09-15T04:12:11.482 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-09-15T04:12:11.482 INFO:tasks.workunit.client.0.smithi138.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-09-15T04:12:11.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-09-15T04:12:12.790 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:12:12.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-09-15T04:12:13.298 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records in 2024-09-15T04:12:13.298 INFO:tasks.workunit.client.0.smithi138.stderr:1024+0 records out 2024-09-15T04:12:13.298 INFO:tasks.workunit.client.0.smithi138.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.495275 s, 2.2 GB/s 2024-09-15T04:12:13.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-09-15T04:12:13.313 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:12:24.080 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-09-15T04:12:24.081 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-09-15T04:12:24.081 INFO:tasks.workunit.client.0.smithi138.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-09-15T04:12:24.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-09-15T04:12:25.391 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:12:25.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:12:25.406 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:12:25.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:12:25.408 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:12:25.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:12:25.428 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:25.428+0000 7f773382a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:25.430 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:25.430+0000 7f773382a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:25.432 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:25.431+0000 7f773382a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:12:25.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:25.764 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:12:25.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:12:25.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:25.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:12:26.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:26.449 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:26.449+0000 7f773382a440 -1 Falling back to public interface 2024-09-15T04:12:27.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:27.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:27.123 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:12:27.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:12:27.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:27.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:12:27.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:28.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:28.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:28.480 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:12:28.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:12:28.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:28.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:12:28.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:29.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:29.824 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:12:29.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:29.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:12:29.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:29.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:12:30.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:31.004 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:31.003+0000 7f773382a440 -1 osd.0 67 log_to_monitors true 2024-09-15T04:12:31.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:31.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:31.200 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:12:31.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:12:31.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:31.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:12:31.625 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 77 up_thru 77 down_at 68 last_clean_interval [56,67) [v2:127.0.0.1:6802/145858,v1:127.0.0.1:6803/145858] [v2:127.0.0.1:6804/145858,v1:127.0.0.1:6805/145858] exists,up cf00fc34-feef-4921-b6dd-6f24bf03a3ef 2024-09-15T04:12:31.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:12:31.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:12:31.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:12:31.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:12:31.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=145858 2024-09-15T04:12:31.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-09-15T04:12:31.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:12:31.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:12:31.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-09-15T04:12:31.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:12:31.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:12:31.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-09-15T04:12:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T04:12:31.637 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:12:31.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:12:31.651 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:31.650+0000 7f1605129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:31.671+0000 7f1605129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:31.672+0000 7f1605129440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:31.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:12:31.993 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:12:31.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:31.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:12:32.187 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:32.187+0000 7f1605129440 -1 Falling back to public interface 2024-09-15T04:12:32.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:33.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:33.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:33.338 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:12:33.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:12:33.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:33.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:12:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:34.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:34.689 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:12:34.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:12:34.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:34.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:12:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:35.299 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:35.298+0000 7f1605129440 -1 osd.1 69 log_to_monitors true 2024-09-15T04:12:36.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:36.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:36.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:12:36.051 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:12:36.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:36.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:12:36.439 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 80 up_thru 80 down_at 70 last_clean_interval [59,69) [v2:127.0.0.1:6810/146849,v1:127.0.0.1:6811/146849] [v2:127.0.0.1:6812/146849,v1:127.0.0.1:6813/146849] exists,up 3ca30242-fa4b-4024-8218-66d11439b6f1 2024-09-15T04:12:36.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:12:36.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:12:36.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:12:36.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=146849 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-09-15T04:12:36.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:12:36.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:12:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:12:36.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-09-15T04:12:36.446 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:12:36.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:12:36.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:36.464+0000 7f5a90229440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:36.466+0000 7f5a90229440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:36.467+0000 7f5a90229440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T04:12:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:12:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:12:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:12:37.240 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:37.240+0000 7f5a90229440 -1 Falling back to public interface 2024-09-15T04:12:37.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:38.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:38.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:38.246 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:12:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:12:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:12:38.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:39.603 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:12:39.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:12:39.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:39.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:12:39.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:40.975 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:12:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:12:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:12:40.997 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:40.997+0000 7f5a90229440 -1 osd.2 72 log_to_monitors true 2024-09-15T04:12:41.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:42.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:42.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:42.466 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:12:42.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:12:42.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:42.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:12:42.878 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 84 up_thru 84 down_at 73 last_clean_interval [62,72) [v2:127.0.0.1:6818/147795,v1:127.0.0.1:6819/147795] [v2:127.0.0.1:6820/147795,v1:127.0.0.1:6821/147795] exists,up d7d4b052-c5b5-4eaa-be71-cec8436c5154 2024-09-15T04:12:42.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:12:42.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:12:42.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:12:42.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=147795 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-09-15T04:12:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:12:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:12:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-09-15T04:12:42.886 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:12:42.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:12:42.904 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:42.903+0000 7f29e5011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:42.919 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:42.919+0000 7f29e5011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:42.920 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:42.920+0000 7f29e5011440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:12:43.250 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:12:43.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:43.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:12:43.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:43.683 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:43.683+0000 7f29e5011440 -1 Falling back to public interface 2024-09-15T04:12:44.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:44.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:44.601 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:12:44.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:12:44.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:44.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:12:44.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:45.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:45.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:45.960 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:12:45.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:12:45.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:45.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:12:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:47.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:47.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:47.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:12:47.311 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:12:47.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:47.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:12:47.596 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:12:47.596+0000 7f29e5011440 -1 osd.3 74 log_to_monitors true 2024-09-15T04:12:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:12:48.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:12:48.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:12:48.717 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:12:48.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:12:48.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:12:48.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:12:49.159 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 88 up_thru 88 down_at 75 last_clean_interval [66,74) [v2:127.0.0.1:6826/148776,v1:127.0.0.1:6827/148776] [v2:127.0.0.1:6828/148776,v1:127.0.0.1:6829/148776] exists,up b2c90f97-5756-450c-81d6-871a4aba7939 2024-09-15T04:12:49.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:12:49.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:12:49.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:12:49.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-09-15T04:12:49.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=148776 2024-09-15T04:12:49.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T04:12:49.187 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:12:49.187 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T04:12:49.187 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_149742 2024-09-15T04:12:49.188 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:12:49.188 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:12:50.187 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 97 81 0.316389 0.316406 0.132836 0.159914 2024-09-15T04:12:51.187 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 242 226 0.441377 0.566406 0.108985 0.138206 2024-09-15T04:12:52.187 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 355 339 0.441374 0.441406 0.163357 0.137301 2024-09-15T04:12:53.187 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 483 467 0.456019 0.5 0.133195 0.136045 2024-09-15T04:12:54.188 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 570 554 0.432777 0.339844 0.173278 0.140066 2024-09-15T04:12:55.188 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 699 683 0.444623 0.503906 0.105176 0.139034 2024-09-15T04:12:56.188 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 812 796 0.444158 0.441406 0.132419 0.140037 2024-09-15T04:12:57.188 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 936 920 0.449181 0.484375 0.127175 0.137749 2024-09-15T04:12:58.188 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 1046 1030 0.447009 0.429688 0.20855 0.138267 2024-09-15T04:12:59.188 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 1146 1130 0.441364 0.390625 0.134011 0.140017 2024-09-15T04:13:00.188 INFO:tasks.workunit.client.0.smithi138.stdout: 11 16 1274 1258 0.446691 0.5 0.130442 0.139569 2024-09-15T04:13:01.188 INFO:tasks.workunit.client.0.smithi138.stdout: 12 16 1405 1389 0.452106 0.511719 0.171973 0.137458 2024-09-15T04:13:02.188 INFO:tasks.workunit.client.0.smithi138.stdout: 13 16 1521 1505 0.452182 0.453125 0.141764 0.137474 2024-09-15T04:13:03.189 INFO:tasks.workunit.client.0.smithi138.stdout: 14 16 1643 1627 0.45392 0.476562 0.117111 0.13705 2024-09-15T04:13:04.189 INFO:tasks.workunit.client.0.smithi138.stdout: 15 16 1755 1739 0.452823 0.4375 0.219101 0.136663 2024-09-15T04:13:05.189 INFO:tasks.workunit.client.0.smithi138.stdout: 16 16 1851 1835 0.447957 0.375 0.161072 0.139071 2024-09-15T04:13:06.189 INFO:tasks.workunit.client.0.smithi138.stdout: 17 16 1960 1944 0.44665 0.425781 0.168464 0.139595 2024-09-15T04:13:07.189 INFO:tasks.workunit.client.0.smithi138.stdout: 18 16 2076 2060 0.447007 0.453125 0.0972929 0.139541 2024-09-15T04:13:08.189 INFO:tasks.workunit.client.0.smithi138.stdout: 19 16 2177 2161 0.444242 0.394531 0.200465 0.13996 2024-09-15T04:13:09.189 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:13:09.189816+0000 min lat: 0.0623128 max lat: 0.44624 avg lat: 0.139119 2024-09-15T04:13:09.189 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:13:09.189 INFO:tasks.workunit.client.0.smithi138.stdout: 20 16 2307 2291 0.447417 0.507812 0.118093 0.139119 2024-09-15T04:13:10.189 INFO:tasks.workunit.client.0.smithi138.stdout: 21 16 2406 2390 0.444525 0.386719 0.133283 0.140164 2024-09-15T04:13:11.189 INFO:tasks.workunit.client.0.smithi138.stdout: 22 16 2518 2502 0.444204 0.4375 0.256376 0.140213 2024-09-15T04:13:12.190 INFO:tasks.workunit.client.0.smithi138.stdout: 23 16 2626 2610 0.443231 0.421875 0.216335 0.140635 2024-09-15T04:13:13.190 INFO:tasks.workunit.client.0.smithi138.stdout: 24 16 2768 2752 0.447873 0.554688 0.0843093 0.139022 2024-09-15T04:13:14.190 INFO:tasks.workunit.client.0.smithi138.stdout: 25 16 2878 2862 0.447144 0.429688 0.229658 0.13927 2024-09-15T04:13:15.190 INFO:tasks.workunit.client.0.smithi138.stdout: 26 16 3016 3000 0.450677 0.539062 0.118626 0.138349 2024-09-15T04:13:16.190 INFO:tasks.workunit.client.0.smithi138.stdout: 27 16 3078 3062 0.442955 0.242188 0.149333 0.140603 2024-09-15T04:13:17.190 INFO:tasks.workunit.client.0.smithi138.stdout: 28 16 3211 3195 0.445688 0.519531 0.0785498 0.139993 2024-09-15T04:13:18.190 INFO:tasks.workunit.client.0.smithi138.stdout: 29 16 3317 3301 0.444596 0.414062 0.149938 0.140085 2024-09-15T04:13:19.190 INFO:tasks.workunit.client.0.smithi138.stdout: 30 16 3447 3431 0.446701 0.507812 0.13468 0.139612 2024-09-15T04:13:20.190 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 30.1416 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 3447 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.446719 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.0720677 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.566406 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.242188 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 114 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 18.4493 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 145 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 62 2024-09-15T04:13:20.191 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.139607 2024-09-15T04:13:20.192 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0501976 2024-09-15T04:13:20.192 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.53601 2024-09-15T04:13:20.192 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0623128 2024-09-15T04:13:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-09-15T04:13:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:13:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:13:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:13:20.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:13:20.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:13:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:13:20.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:13:20.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=330712481804 2024-09-15T04:13:20.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 330712481804 2024-09-15T04:13:20.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-330712481804' 2024-09-15T04:13:20.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:13:20.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:13:20.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=343597383692 2024-09-15T04:13:20.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 343597383692 2024-09-15T04:13:20.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-330712481804 1-343597383692' 2024-09-15T04:13:20.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:13:20.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:13:21.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=360777252874 2024-09-15T04:13:21.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 360777252874 2024-09-15T04:13:21.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-330712481804 1-343597383692 2-360777252874' 2024-09-15T04:13:21.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:13:21.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:13:21.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=377957122057 2024-09-15T04:13:21.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 377957122057 2024-09-15T04:13:21.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-330712481804 1-343597383692 2-360777252874 3-377957122057' 2024-09-15T04:13:21.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:13:21.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-330712481804 2024-09-15T04:13:21.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:13:21.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:13:21.134 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-330712481804 2024-09-15T04:13:21.134 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:13:21.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=330712481804 2024-09-15T04:13:21.135 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 330712481804 2024-09-15T04:13:21.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 330712481804' 2024-09-15T04:13:21.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:13:21.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 330712481805 -lt 330712481804 2024-09-15T04:13:21.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:13:21.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-343597383692 2024-09-15T04:13:21.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:13:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:13:21.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-343597383692 2024-09-15T04:13:21.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:13:21.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=343597383692 2024-09-15T04:13:21.494 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 343597383692 2024-09-15T04:13:21.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 343597383692' 2024-09-15T04:13:21.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:13:21.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 343597383692 -lt 343597383692 2024-09-15T04:13:21.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:13:21.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-360777252874 2024-09-15T04:13:21.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:13:21.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:13:21.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-360777252874 2024-09-15T04:13:21.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:13:21.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=360777252874 2024-09-15T04:13:21.872 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 360777252874 2024-09-15T04:13:21.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 360777252874' 2024-09-15T04:13:21.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:13:22.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 360777252875 -lt 360777252874 2024-09-15T04:13:22.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:13:22.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-377957122057 2024-09-15T04:13:22.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:13:22.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:13:22.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-377957122057 2024-09-15T04:13:22.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:13:22.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=377957122057 2024-09-15T04:13:22.223 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 377957122057 2024-09-15T04:13:22.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 377957122057' 2024-09-15T04:13:22.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:13:22.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 377957122057 -lt 377957122057 2024-09-15T04:13:22.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:13:22.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:13:22.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:13:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T04:13:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:13:23.038 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:13:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:13:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:13:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:13:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:13:23.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:13:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T04:13:23.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:13:23.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:13:23.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:13:23.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:13:23.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:13:23.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:13:23.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:13:23.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:13:23.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:13:23.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:13:23.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:13:23.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:13:23.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:13:23.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:13:23.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:13:23.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:13:23.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:13:23.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:13:23.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:13:23.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:13:23.959 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:13:23.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:13:23.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:13:23.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:13:24.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:13:24.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:24.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:13:24.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:13:24.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:13:24.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:13:24.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:13:24.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:13:24.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:13:24.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:13:24.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:13:24.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:13:24.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:13:24.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:13:24.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:13:24.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:13:24.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:13:24.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:13:24.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:24.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:24.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:13:24.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:13:24.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:13:24.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T04:13:24.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:13:24.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:24.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:24.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102313 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-09-15T04:13:24.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-09-15T04:13:24.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-09-15T04:13:24.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-09-15T04:13:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none ' 2024-09-15T04:13:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T04:13:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-15T04:13:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T04:13:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T04:13:24.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:24.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:13:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:13:25.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:13:25.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:13:25.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get fsid 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:13:25.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:13:25.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get mon_host 2024-09-15T04:13:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T04:13:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:26.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:13:26.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:13:26.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:13:26.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:13:26.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:13:26.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:13:26.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8851782e-1a60-4b87-acb0-a48dc04bbc7e 2024-09-15T04:13:26.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8851782e-1a60-4b87-acb0-a48dc04bbc7e' 2024-09-15T04:13:26.133 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 8851782e-1a60-4b87-acb0-a48dc04bbc7e 2024-09-15T04:13:26.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:13:26.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDmXuZm/8PKCBAAQuqf5LMeBlHZUZs3HDja+A== 2024-09-15T04:13:26.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDmXuZm/8PKCBAAQuqf5LMeBlHZUZs3HDja+A=="}' 2024-09-15T04:13:26.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8851782e-1a60-4b87-acb0-a48dc04bbc7e -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:13:26.435 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:13:26.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:13:26.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDmXuZm/8PKCBAAQuqf5LMeBlHZUZs3HDja+A== --osd-uuid 8851782e-1a60-4b87-acb0-a48dc04bbc7e 2024-09-15T04:13:26.474 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.473+0000 7f4d5b68b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:26.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.481+0000 7f4d5b68b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:26.482 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.482+0000 7f4d5b68b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:26.482 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.482+0000 7f4d5b68b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:13:26.483 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.482+0000 7f4d5b68b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:13:26.483 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.482+0000 7f4d5b68b440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:13:26.513 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:26.512+0000 7f4d5b68b440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T04:13:33.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T04:13:33.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:13:33.844 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:13:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:13:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:13:34.327 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:13:34.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:13:34.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:13:34.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:13:34.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:13:34.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:13:34.354 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:34.354+0000 7f255f384440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:34.356 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:34.356+0000 7f255f384440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:34.357 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:34.357+0000 7f255f384440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:13:34.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:34.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:13:34.682 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:13:34.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:34.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:35.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:35.121 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:35.121+0000 7f255f384440 -1 Falling back to public interface 2024-09-15T04:13:36.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:36.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:36.042 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:13:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:13:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:36.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:36.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:37.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:37.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:37.396 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:13:37.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:13:37.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:37.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:37.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:38.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:38.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:38.747 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:13:38.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:13:38.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:38.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:39.004 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:39.004+0000 7f255f384440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:13:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:40.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:40.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:40.109 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:13:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:13:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:41.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:42.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:42.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:42.320 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:13:42.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:13:42.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:42.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:43.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:13:43.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:13:43.515+0000 7f255f384440 -1 bdev(0x55a338af6800 td/osd-bluefs-volume-ops/0/block) aio_submit retries 8 2024-09-15T04:13:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:13:44.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:13:44.386 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:13:44.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:13:44.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:13:44.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:13:44.791 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/151630,v1:127.0.0.1:6803/151630] [v2:127.0.0.1:6804/151630,v1:127.0.0.1:6805/151630] exists,up 8851782e-1a60-4b87-acb0-a48dc04bbc7e 2024-09-15T04:13:44.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:13:44.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:13:44.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:13:44.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:13:44.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=151630 2024-09-15T04:13:44.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-09-15T04:13:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-09-15T04:13:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-15T04:13:50.377 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T04:13:50.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:13:51.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-09-15T04:13:51.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-09-15T04:13:51.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-09-15T04:13:51.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-09-15T04:13:51.422 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:13:51.424 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-09-15T04:13:51.424 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_152928 2024-09-15T04:13:51.454 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:13:51.455 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:13:52.455 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 17 1 3.99955 4 0.101028 0.101028 2024-09-15T04:13:53.455 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 49 33 65.9913 128 0.533842 0.772246 2024-09-15T04:13:54.455 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 81 65 86.6561 128 0.74314 0.673219 2024-09-15T04:13:55.455 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 112 96 95.9868 124 0.576705 0.654385 2024-09-15T04:13:56.455 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 129 113 90.3886 68 0.617475 0.647344 2024-09-15T04:13:57.455 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 145 129 85.9889 64 0.926689 0.659737 2024-09-15T04:13:58.455 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 161 145 82.8468 64 0.983924 0.693 2024-09-15T04:13:59.455 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 161 145 72.4906 0 - 0.693 2024-09-15T04:14:00.456 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 177 161 71.5463 32 1.80164 0.811623 2024-09-15T04:14:01.456 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 177 161 64.3915 0 - 0.811623 2024-09-15T04:14:02.456 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 10.2936 2024-09-15T04:14:02.456 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 177 2024-09-15T04:14:02.456 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4194304 2024-09-15T04:14:02.456 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4194304 2024-09-15T04:14:02.456 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 68.7803 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 52.2235 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 128 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 17 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 13.0559 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 32 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 0 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.906729 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.48588 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 1.9234 2024-09-15T04:14:02.457 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.101028 2024-09-15T04:14:02.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-09-15T04:14:12.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T04:14:12.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-09-15T04:14:12.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1072693248 2024-09-15T04:14:12.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-09-15T04:14:12.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-09-15T04:14:13.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=883294208 2024-09-15T04:14:13.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-09-15T04:14:13.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 883294208 -eq 0 2024-09-15T04:14:13.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-09-15T04:14:13.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 883294208 -gt 0 2024-09-15T04:14:13.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151630 2024-09-15T04:14:13.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-09-15T04:14:14.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 151630 2024-09-15T04:14:14.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (151630) - No such process 2024-09-15T04:14:14.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-09-15T04:14:14.449 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:14:14.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-09-15T04:14:14.478 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:14:24.745 INFO:tasks.workunit.client.0.smithi138.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-09-15T04:14:24.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-09-15T04:14:24.836 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:14:26.104 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x100000000 : using 0x74b79000(1.8 GiB) 2024-09-15T04:14:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-09-15T04:14:27.652 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:14:27.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:14:27.688 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:14:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:14:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:14:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:14:27.690 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:14:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:14:27.714 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:27.713+0000 7fb6ecc1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:27.715 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:27.715+0000 7fb6ecc1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:27.716 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:27.716+0000 7fb6ecc1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:14:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:28.040 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:14:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:14:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:28.230 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:28.230+0000 7fb6ecc1c440 -1 Falling back to public interface 2024-09-15T04:14:28.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:29.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:29.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:29.470 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:14:29.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:14:29.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:29.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:29.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:30.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:30.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:30.830 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:14:30.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:14:30.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:30.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:31.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:32.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:32.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:32.190 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:14:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:14:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:32.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:33.756 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:14:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:14:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:34.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:35.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:35.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:35.172 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:14:35.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:14:35.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:35.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:35.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:36.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:36.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:36.594 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:14:36.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:14:36.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:36.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:36.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:36.725+0000 7fb6ecc1c440 -1 osd.0 10 log_to_monitors true 2024-09-15T04:14:37.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:38.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:38.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:38.045 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:14:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:14:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:39.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:40.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:40.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:40.627 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:14:40.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:14:40.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:40.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:42.104 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:42.103+0000 7fb6ecc1c440 -1 bdev(0x5610fa68e800 td/osd-bluefs-volume-ops/0/block) aio_submit retries 9 2024-09-15T04:14:42.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:43.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:14:43.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:43.398 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:14:43.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:14:43.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:43.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:44.198 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 13 up_thru 7 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/154161,v1:127.0.0.1:6803/154161] [v2:127.0.0.1:6804/154161,v1:127.0.0.1:6805/154161] exists,up 8851782e-1a60-4b87-acb0-a48dc04bbc7e 2024-09-15T04:14:44.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:14:44.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:14:44.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:14:44.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:14:44.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=154161 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:14:44.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:14:44.203 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:14:44.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:14:44.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:14:44.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:14:44.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:14:44.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T04:14:44.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:14:44.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:14:44.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:14:44.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574852 2024-09-15T04:14:44.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574852 2024-09-15T04:14:44.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-55834574852' 2024-09-15T04:14:44.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:14:44.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-55834574852 2024-09-15T04:14:44.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:14:44.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:14:44.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-55834574852 2024-09-15T04:14:44.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:14:44.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574852 2024-09-15T04:14:44.931 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 55834574852 2024-09-15T04:14:44.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-15T04:14:44.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:14:45.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 2 -lt 55834574852 2024-09-15T04:14:45.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:14:46.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:14:46.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:14:46.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-15T04:14:46.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:14:46.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:14:46.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:14:47.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T04:14:47.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:14:47.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:14:47.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:14:47.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:14:47.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:14:47.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:14:47.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:14:47.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T04:14:47.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:14:47.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:14:47.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:14:47.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:14:47.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:14:47.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:14:47.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:14:47.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:14:48.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:14:48.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:14:48.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:14:48.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:14:48.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:14:48.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:14:48.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:14:48.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:14:48.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:14:48.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:14:48.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:14:48.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:14:48.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:14:48.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:14:48.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:14:48.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-09-15T04:14:48.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:14:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:14:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:14:48.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:14:48.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:14:48.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:14:48.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:14:48.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:14:48.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:14:48.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:14:48.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:14:48.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:14:48.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:14:48.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:14:48.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:14:48.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:14:48.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:14:48.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:14:48.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:14:48.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-09-15T04:14:48.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:14:48.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.102313 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-09-15T04:14:48.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-09-15T04:14:48.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-09-15T04:14:48.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-09-15T04:14:48.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-09-15T04:14:48.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-09-15T04:14:48.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-09-15T04:14:48.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-09-15T04:14:48.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:48.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:48.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:14:48.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:14:48.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:14:48.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:14:48.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:14:48.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:14:48.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get fsid 2024-09-15T04:14:48.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.102313/ceph-mon.a.asok 2024-09-15T04:14:48.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:14:48.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.102313/ceph-mon.a.asok config get mon_host 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-09-15T04:14:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:14:49.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:14:49.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:14:49.943 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:14:49.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd7d47e4-f979-4369-b68e-696f9317f1c6 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dd7d47e4-f979-4369-b68e-696f9317f1c6' 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 dd7d47e4-f979-4369-b68e-696f9317f1c6 2024-09-15T04:14:49.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:14:49.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA5X+ZmPE9sORAAUaP200Q+SHUe5DL9JXt9KQ== 2024-09-15T04:14:49.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA5X+ZmPE9sORAAUaP200Q+SHUe5DL9JXt9KQ=="}' 2024-09-15T04:14:49.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd7d47e4-f979-4369-b68e-696f9317f1c6 -i td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:14:50.260 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:14:50.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-09-15T04:14:50.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA5X+ZmPE9sORAAUaP200Q+SHUe5DL9JXt9KQ== --osd-uuid dd7d47e4-f979-4369-b68e-696f9317f1c6 2024-09-15T04:14:50.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.305+0000 7fbe63786440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:50.307 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.307+0000 7fbe63786440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:50.308 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.308+0000 7fbe63786440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:50.309 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.308+0000 7fbe63786440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:14:50.309 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.308+0000 7fbe63786440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:14:50.309 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.308+0000 7fbe63786440 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-09-15T04:14:50.338 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:50.338+0000 7fbe63786440 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-09-15T04:14:57.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-09-15T04:14:57.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:14:57.930 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:14:57.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:14:57.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:14:58.423 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:14:58.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:14:58.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:14:58.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:14:58.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:14:58.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:14:58.457 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:58.456+0000 7f04da618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:58.458 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:58.458+0000 7f04da618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:58.459 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:58.459+0000 7f04da618440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:14:58.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:14:58.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:14:58.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:14:58.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:14:58.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:14:58.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:14:58.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:14:58.824 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:14:58.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:14:58.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:14:59.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:14:59.242 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:14:59.242+0000 7f04da618440 -1 Falling back to public interface 2024-09-15T04:15:00.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:00.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:00.192 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:15:00.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:15:00.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:00.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:00.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:15:01.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:01.553 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:15:01.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:01.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:15:01.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:01.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:01.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:15:02.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:02.907 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:15:02.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:02.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:15:02.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:02.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:03.145 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:15:03.144+0000 7f04da618440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:15:03.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:15:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:04.272 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:15:04.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:15:04.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:04.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:05.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:15:06.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:06.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:06.486 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:15:06.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:15:06.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:06.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:07.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:15:08.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:15:08.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:15:08.134 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:15:08.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:15:08.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:15:08.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:15:08.542 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/156961,v1:127.0.0.1:6803/156961] [v2:127.0.0.1:6804/156961,v1:127.0.0.1:6805/156961] exists,up dd7d47e4-f979-4369-b68e-696f9317f1c6 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=156961 2024-09-15T04:15:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-09-15T04:15:13.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-09-15T04:15:13.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-15T04:15:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T04:15:14.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:15:15.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-09-15T04:15:15.133 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:15:15.133 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-09-15T04:15:15.133 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_158004 2024-09-15T04:15:15.133 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:15:15.133 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:15:16.133 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 128 112 0.437446 0.4375 0.142518 0.13174 2024-09-15T04:15:17.133 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 272 256 0.499928 0.5625 0.100588 0.123832 2024-09-15T04:15:18.133 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 400 384 0.499926 0.5 0.117628 0.121517 2024-09-15T04:15:19.133 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 560 544 0.531175 0.625 0.0922082 0.11744 2024-09-15T04:15:20.134 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 674 658 0.513991 0.445312 0.121682 0.118396 2024-09-15T04:15:21.134 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 818 802 0.522062 0.5625 0.108892 0.118508 2024-09-15T04:15:22.134 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 946 930 0.5189 0.5 0.128631 0.118204 2024-09-15T04:15:23.134 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 1090 1074 0.52434 0.5625 0.117143 0.118174 2024-09-15T04:15:24.134 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 1218 1202 0.521628 0.5 0.142128 0.11791 2024-09-15T04:15:25.134 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 1314 1298 0.506959 0.375 0.125475 0.122134 2024-09-15T04:15:26.134 INFO:tasks.workunit.client.0.smithi138.stdout: 11 16 1440 1424 0.50561 0.492188 0.159365 0.122737 2024-09-15T04:15:27.135 INFO:tasks.workunit.client.0.smithi138.stdout: 12 16 1586 1570 0.510995 0.570312 0.108787 0.121268 2024-09-15T04:15:28.135 INFO:tasks.workunit.client.0.smithi138.stdout: 13 16 1714 1698 0.510145 0.5 0.143417 0.121073 2024-09-15T04:15:29.135 INFO:tasks.workunit.client.0.smithi138.stdout: 14 16 1856 1840 0.513321 0.554688 0.117308 0.121541 2024-09-15T04:15:30.135 INFO:tasks.workunit.client.0.smithi138.stdout: 15 16 1970 1954 0.508784 0.445312 0.200755 0.12146 2024-09-15T04:15:31.135 INFO:tasks.workunit.client.0.smithi138.stdout: 16 16 2130 2114 0.516041 0.625 0.117248 0.120557 2024-09-15T04:15:32.135 INFO:tasks.workunit.client.0.smithi138.stdout: 17 16 2256 2240 0.514633 0.492188 0.159243 0.120935 2024-09-15T04:15:33.136 INFO:tasks.workunit.client.0.smithi138.stdout: 18 16 2386 2370 0.51425 0.507812 0.109128 0.120917 2024-09-15T04:15:34.135 INFO:tasks.workunit.client.0.smithi138.stdout: 19 16 2528 2512 0.516375 0.554688 0.167234 0.120903 2024-09-15T04:15:35.136 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:15:35.136270+0000 min lat: 0.0630062 max lat: 0.375678 avg lat: 0.119967 2024-09-15T04:15:35.136 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:15:35.136 INFO:tasks.workunit.client.0.smithi138.stdout: 20 16 2674 2658 0.519068 0.570312 0.100675 0.119967 2024-09-15T04:15:36.136 INFO:tasks.workunit.client.0.smithi138.stdout: 21 16 2800 2784 0.517785 0.492188 0.158979 0.120214 2024-09-15T04:15:37.136 INFO:tasks.workunit.client.0.smithi138.stdout: 22 16 2930 2914 0.517329 0.507812 0.100613 0.120369 2024-09-15T04:15:38.136 INFO:tasks.workunit.client.0.smithi138.stdout: 23 16 3072 3056 0.51895 0.554688 0.167566 0.120244 2024-09-15T04:15:39.136 INFO:tasks.workunit.client.0.smithi138.stdout: 24 16 3216 3200 0.520761 0.5625 0.0839637 0.119856 2024-09-15T04:15:40.136 INFO:tasks.workunit.client.0.smithi138.stdout: 25 16 3344 3328 0.519928 0.5 0.146136 0.12006 2024-09-15T04:15:41.137 INFO:tasks.workunit.client.0.smithi138.stdout: 26 16 3472 3456 0.519159 0.5 0.134031 0.120163 2024-09-15T04:15:42.137 INFO:tasks.workunit.client.0.smithi138.stdout: 27 16 3602 3586 0.518736 0.507812 0.133705 0.119949 2024-09-15T04:15:43.137 INFO:tasks.workunit.client.0.smithi138.stdout: 28 16 3760 3744 0.522249 0.617188 0.117032 0.119576 2024-09-15T04:15:44.137 INFO:tasks.workunit.client.0.smithi138.stdout: 29 16 3874 3858 0.519593 0.445312 0.159148 0.119997 2024-09-15T04:15:45.137 INFO:tasks.workunit.client.0.smithi138.stdout: 30 16 4002 3986 0.518938 0.5 0.117388 0.120009 2024-09-15T04:15:46.137 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 30.0567 2024-09-15T04:15:46.137 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 4002 2024-09-15T04:15:46.137 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.52011 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.057982 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.625 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.375 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 133 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 14.8434 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 160 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 96 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.119989 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0308121 2024-09-15T04:15:46.138 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.375678 2024-09-15T04:15:46.139 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0630062 2024-09-15T04:15:46.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-09-15T04:15:51.178 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-15T04:15:51.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-15T04:15:51.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-09-15T04:15:51.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-15T04:15:51.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-15T04:15:51.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-09-15T04:15:51.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-09-15T04:15:51.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-09-15T04:15:51.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-15T04:15:51.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4278472704 2024-09-15T04:15:51.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156961 2024-09-15T04:15:51.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-09-15T04:15:52.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 156961 2024-09-15T04:15:52.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (156961) - No such process 2024-09-15T04:15:52.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-09-15T04:15:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:15:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-09-15T04:15:52.872 INFO:tasks.workunit.client.0.smithi138.stdout:allocmap bluestore.allocmap 2024-09-15T04:15:55.634 INFO:tasks.workunit.client.0.smithi138.stdout:allocmap success 2024-09-15T04:15:55.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:15:58.933 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:15:58.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-09-15T04:15:58.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-09-15T04:15:58.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-09-15T04:15:58.970 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:16:01.094 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0x12b00000(299 MiB) 2024-09-15T04:16:01.094 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x100000000 : using 0xfbb000(16 MiB) 2024-09-15T04:16:01.094 INFO:tasks.workunit.client.0.smithi138.stdout:Expanding DB/WAL... 2024-09-15T04:16:01.094 INFO:tasks.workunit.client.0.smithi138.stdout:1 : expanding to 0x1073741824 2024-09-15T04:16:01.129 INFO:tasks.workunit.client.0.smithi138.stdout:1 : size label updated to 1073741824 2024-09-15T04:16:01.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-09-15T04:16:04.531 INFO:tasks.workunit.client.0.smithi138.stdout:fsck success 2024-09-15T04:16:04.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-15T04:16:04.569 INFO:tasks.workunit.client.0.smithi138.stdout:qfsck bluestore.quick-fsck 2024-09-15T04:16:07.331 INFO:tasks.workunit.client.0.smithi138.stdout:qfsck success 2024-09-15T04:16:07.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-09-15T04:16:07.364 INFO:tasks.workunit.client.0.smithi138.stdout:inferring bluefs devices from bluestore path 2024-09-15T04:16:10.129 INFO:tasks.workunit.client.0.smithi138.stdout:1 : device size 0x3fffe000 : using 0x12b00000(299 MiB) 2024-09-15T04:16:10.129 INFO:tasks.workunit.client.0.smithi138.stdout:2 : device size 0x100000000 : using 0xfbb000(16 MiB) 2024-09-15T04:16:10.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-09-15T04:16:10.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:16:10.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-09-15T04:16:10.139 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:16:10.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:16:10.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=7d52180a-cd81-4625-8637-b0e9a8070056 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=37edfab8-5f4c-4a35-9770-b623e589e9d3 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=52040b91-cf00-4e25-b879-faa0971e63fe --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.102313/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:16:10.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-09-15T04:16:10.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:16:10.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:16:10.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:16:10.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:16:10.171 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:10.168+0000 7f2c1e01b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:10.172 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:10.170+0000 7f2c1e01b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:10.173 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:10.171+0000 7f2c1e01b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:16:10.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:10.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:16:10.499 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:16:10.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:10.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:10.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:10.937 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:10.936+0000 7f2c1e01b440 -1 Falling back to public interface 2024-09-15T04:16:11.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:11.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:11.856 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:16:11.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:16:11.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:11.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:12.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:13.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:13.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:13.258 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:16:13.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:16:13.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:13.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:13.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:14.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:14.630 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:16:14.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:14.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:16:14.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:14.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:15.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:15.777 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:15.776+0000 7f2c1e01b440 -1 osd.0 10 log_to_monitors true 2024-09-15T04:16:16.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:16.010 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:16:16.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:16.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:16:16.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:16.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:16.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:17.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:17.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:17.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:16:17.879 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:16:17.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:17.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:19.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:20.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:20.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:20.123 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:16:20.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:16:20.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:20.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:20.496 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/160379,v1:127.0.0.1:6803/160379] [v2:127.0.0.1:6804/160379,v1:127.0.0.1:6805/160379] exists,up dd7d47e4-f979-4369-b68e-696f9317f1c6 2024-09-15T04:16:20.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:16:20.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:16:20.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:16:20.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-09-15T04:16:20.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=160379 2024-09-15T04:16:20.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:16:20.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:16:20.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:16:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T04:16:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:16:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:16:20.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574852 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574852 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-55834574852' 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-55834574852 2024-09-15T04:16:21.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:16:21.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:16:21.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-55834574852 2024-09-15T04:16:21.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:16:21.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574852 2024-09-15T04:16:21.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-09-15T04:16:21.102 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 55834574852 2024-09-15T04:16:21.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:16:21.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 55834574852 2024-09-15T04:16:21.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:16:22.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:16:22.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:16:22.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574852 -lt 55834574852 2024-09-15T04:16:22.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:16:22.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:16:22.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:16:23.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:16:23.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T04:16:23.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:16:23.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:16:23.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:16:24.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T04:16:24.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:16:24.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:16:24.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-09-15T04:16:24.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-09-15T04:16:24.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-09-15T04:16:24.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4278427648 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-45056 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -45056)' 2024-09-15T04:16:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 45056 2024-09-15T04:16:24.459 INFO:tasks.workunit.client.0.smithi138.stdout:45056 2024-09-15T04:16:24.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 45056 -gt 131072 ']' 2024-09-15T04:16:24.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 160379 2024-09-15T04:16:24.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-09-15T04:16:25.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 160379 2024-09-15T04:16:25.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (160379) - No such process 2024-09-15T04:16:25.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-09-15T04:16:25.864 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:16:25.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-09-15T04:16:25.898 INFO:tasks.workunit.client.0.smithi138.stdout:qfsck bluestore.quick-fsck 2024-09-15T04:16:28.912 INFO:tasks.workunit.client.0.smithi138.stdout:qfsck success 2024-09-15T04:16:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-09-15T04:16:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:16:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:16:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:16:28.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:16:28.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:16:28.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:16:28.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:16:28.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:16:29.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:16:29.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:16:29.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:16:29.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:16:29.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:16:29.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:16:29.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:16:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:16:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:16:29.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:16:29.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:16:29.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:16:29.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:16:29.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:16:29.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:16:29.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-bluefs-volume-ops 0 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:16:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:16:29.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:16:29.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:16:29.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:16:29.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:16:29.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:16:29.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:16:29.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:16:29.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:16:29.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:16:29.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:16:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:16:29.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-09-15T04:16:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:16:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.102313 2024-09-15T04:16:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.102313 2024-09-15T04:16:29.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:16:29.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:16:29.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:16:29.247 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:16:29.247 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:16:29.274 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-09-15T04:16:29.274 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-09-15T04:16:29.338 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:16:29.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-config 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-09-15T04:16:29.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-09-15T04:16:29.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none ' 2024-09-15T04:16:29.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-15T04:16:29.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-09-15T04:16:29.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:16:29.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_config_track 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_default_adjustment' 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:16:29.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:16:29.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:16:29.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:16:29.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:16:29.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:16:29.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:16:29.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:16:29.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:16:29.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:16:29.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:16:29.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:29.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:16:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:16:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:16:29.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-15T04:16:29.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:16:29.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:29.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162152 2024-09-15T04:16:29.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-15T04:16:29.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-15T04:16:29.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:16:29.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:16:29.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:16:29.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:16:29.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:29.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:16:29.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:16:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-mon.a.asok 2024-09-15T04:16:29.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:16:29.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162152/ceph-mon.a.asok config get fsid 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-mon.a.asok 2024-09-15T04:16:30.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:16:30.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162152/ceph-mon.a.asok config get mon_host 2024-09-15T04:16:30.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-15T04:16:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:30.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:16:30.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:16:30.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-15T04:16:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T04:16:31.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:16:31.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:16:31.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:16:31.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-15T04:16:31.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-15T04:16:31.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:16:31.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:16:31.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:16:31.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-15T04:16:31.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:16:31.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=432cc33b-2c1f-40f5-9ed3-dfd794e64b3e 2024-09-15T04:16:31.017 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 432cc33b-2c1f-40f5-9ed3-dfd794e64b3e 2024-09-15T04:16:31.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 432cc33b-2c1f-40f5-9ed3-dfd794e64b3e' 2024-09-15T04:16:31.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:16:31.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfX+ZmHEh7ARAAMIK7JHJyBHic7Mvm5Dlawg== 2024-09-15T04:16:31.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfX+ZmHEh7ARAAMIK7JHJyBHic7Mvm5Dlawg=="}' 2024-09-15T04:16:31.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 432cc33b-2c1f-40f5-9ed3-dfd794e64b3e -i td/osd-config/0/new.json 2024-09-15T04:16:31.321 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:16:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-15T04:16:31.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQCfX+ZmHEh7ARAAMIK7JHJyBHic7Mvm5Dlawg== --osd-uuid 432cc33b-2c1f-40f5-9ed3-dfd794e64b3e 2024-09-15T04:16:31.355 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.355+0000 7f0561b4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:31.368 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.367+0000 7f0561b4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:31.369 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.368+0000 7f0561b4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:31.369 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.369+0000 7f0561b4a440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:16:31.369 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.369+0000 7f0561b4a440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:16:31.369 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.369+0000 7f0561b4a440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:16:31.408 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:31.407+0000 7f0561b4a440 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-15T04:16:36.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-15T04:16:36.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:16:36.826 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:16:36.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:16:36.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:16:37.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:16:37.301 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:16:37.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-09-15T04:16:37.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:16:37.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:16:37.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:16:37.322 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:37.322+0000 7f693f32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:37.324 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:37.324+0000 7f693f32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:37.325 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:37.325+0000 7f693f32c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:16:37.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:37.646 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:16:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:16:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:37.846 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:37.846+0000 7f693f32c440 -1 Falling back to public interface 2024-09-15T04:16:38.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:39.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:39.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:39.021 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:16:39.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:16:39.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:39.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:39.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:16:40.308+0000 7f693f32c440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:16:40.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:40.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:40.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:16:40.378 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:16:40.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:40.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:40.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:41.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:41.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:41.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:16:41.909 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:16:41.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:41.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:42.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:43.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:43.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:43.331 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:16:43.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:16:43.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:43.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:43.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:44.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:44.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:44.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:16:44.747 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:16:44.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:44.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:45.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:46.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:46.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:46.177 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:16:46.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:16:46.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:46.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:46.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:47.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:47.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:47.559 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:16:47.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:16:47.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:47.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:48.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:49.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:49.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:49.006 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:16:49.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:16:49.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:49.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:49.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:50.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:50.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:50.384 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:16:50.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:16:50.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:50.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:50.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:51.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:51.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:51.743 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:16:51.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:16:51.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:51.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:52.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:53.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:53.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:53.101 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:16:53.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:16:53.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:53.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:53.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:54.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:54.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:54.974 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:16:54.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:16:54.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:54.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:55.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:56.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:56.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:56.743 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:16:56.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:16:56.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:56.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:57.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:16:58.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:16:58.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:16:58.573 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:16:58.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:16:58.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:16:58.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:16:59.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:00.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:00.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:00.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:17:00.045 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:17:00.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:00.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:00.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:01.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:01.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:01.734 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:17:01.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:17:01.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:01.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:02.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:03.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:03.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:03.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:17:03.486 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:17:03.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:03.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:03.862 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/163243,v1:127.0.0.1:6803/163243] [v2:127.0.0.1:6804/163243,v1:127.0.0.1:6805/163243] exists,up 432cc33b-2c1f-40f5-9ed3-dfd794e64b3e 2024-09-15T04:17:03.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:17:03.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:17:03.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:17:03.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-09-15T04:17:03.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-osd.0.asok 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-09-15T04:17:03.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.162152/ceph-osd.0.asok log flush 2024-09-15T04:17:03.938 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:17:03.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:16:40.614+0000 7f693f32c440 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:16:43.308+0000 7f692e97b640 10 log_client will send 2024-09-15T04:16:40.615563+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:16:43.626+0000 7f692f97d640 10 log_client logged 2024-09-15T04:16:40.615563+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:17:03.284+0000 7f693a5f7640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:17:03.311+0000 7f692e97b640 10 log_client will send 2024-09-15T04:17:03.285041+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.947 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:17:03.579+0000 7f692f97d640 10 log_client logged 2024-09-15T04:17:03.285041+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:03.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:03.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:03.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:03.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:04.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:04.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:04.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:04.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:04.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:04.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:04.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:04.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:04.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:04.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:04.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:04.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:04.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:04.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-15T04:17:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:04.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:04.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:04.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:04.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:04.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:04.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:04.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:04.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:04.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:04.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:04.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:04.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:04.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:04.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:04.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:04.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:04.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:04.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:04.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-15T04:17:04.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:17:04.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162152 2024-09-15T04:17:04.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:17:04.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-09-15T04:17:04.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-09-15T04:17:04.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:17:04.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:04.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:04.437 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:04.437 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.438 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:17:04.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:17:04.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:17:04.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:17:04.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:17:04.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-mon.a.asok 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:17:04.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162152/ceph-mon.a.asok config get fsid 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:04.786 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:04.787 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:04.787 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-mon.a.asok 2024-09-15T04:17:04.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:17:04.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162152/ceph-mon.a.asok config get mon_host 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-09-15T04:17:04.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:05.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:05.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:17:05.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:17:05.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T04:17:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:17:05.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:17:05.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f5af073-8415-4652-88e7-6160d4ed8177 2024-09-15T04:17:05.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9f5af073-8415-4652-88e7-6160d4ed8177' 2024-09-15T04:17:05.721 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 9f5af073-8415-4652-88e7-6160d4ed8177 2024-09-15T04:17:05.721 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:17:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBX+Zm5SWEKxAAasyWC4+WLipltVuVa1VHoA== 2024-09-15T04:17:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBX+Zm5SWEKxAAasyWC4+WLipltVuVa1VHoA=="}' 2024-09-15T04:17:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f5af073-8415-4652-88e7-6160d4ed8177 -i td/osd-config/0/new.json 2024-09-15T04:17:06.043 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:17:06.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-09-15T04:17:06.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBX+Zm5SWEKxAAasyWC4+WLipltVuVa1VHoA== --osd-uuid 9f5af073-8415-4652-88e7-6160d4ed8177 2024-09-15T04:17:06.076 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.075+0000 7fe56391e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:06.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.089+0000 7fe56391e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.090+0000 7fe56391e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.090+0000 7fe56391e440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:17:06.091 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.090+0000 7fe56391e440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:17:06.091 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.090+0000 7fe56391e440 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-09-15T04:17:06.129 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:06.129+0000 7fe56391e440 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-09-15T04:17:11.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-09-15T04:17:11.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:17:11.670 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:17:11.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:17:11.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:17:12.149 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:17:12.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:17:12.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=7fb7c572-0318-4bae-9d1a-eed3f9222d42 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.162152/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:17:12.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:17:12.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:17:12.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:17:12.170 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:12.169+0000 7f0810a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:12.172 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:12.171+0000 7f0810a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:12.173 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:12.172+0000 7f0810a09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:12.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:17:12.495 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:17:12.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:12.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:12.685 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:12.685+0000 7f0810a09440 -1 Falling back to public interface 2024-09-15T04:17:12.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:13.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:13.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:13.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:17:13.851 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:17:13.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:13.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:14.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:15.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:15.155+0000 7f0810a09440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:17:15.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:15.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:15.203 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:17:15.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:17:15.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:15.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:15.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:16.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:16.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:16.725 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:17:16.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:17:16.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:16.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:17.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:18.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:18.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:18.137 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:17:18.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:17:18.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:18.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:18.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:19.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:19.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:19.511 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:17:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:17:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:21.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:21.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:21.030 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:17:21.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:17:21.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:21.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:21.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:22.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:22.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:22.406 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:17:22.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:17:22.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:22.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:22.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:23.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:23.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:23.855 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:17:23.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:17:23.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:23.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:24.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:25.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:25.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:25.243 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:17:25.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:17:25.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:25.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:25.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:26.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:26.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:26.632 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:17:26.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:17:26.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:26.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:27.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:28.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:28.528 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:17:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:17:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:29.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:30.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:30.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:30.045 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:17:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:17:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:30.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:31.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:31.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:31.833 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:17:31.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:17:31.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:31.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:32.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:33.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:17:33.418 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:17:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:33.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:35.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:35.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:35.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:17:35.740 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:17:35.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:35.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:37.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:38.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:38.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:38.037 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:17:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:17:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:38.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:38.419 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/165268,v1:127.0.0.1:6803/165268] [v2:127.0.0.1:6804/165268,v1:127.0.0.1:6805/165268] exists,up 9f5af073-8415-4652-88e7-6160d4ed8177 2024-09-15T04:17:38.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:17:38.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:17:38.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:17:38.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-09-15T04:17:38.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-09-15T04:17:38.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-09-15T04:17:38.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-09-15T04:17:38.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-09-15T04:17:38.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-09-15T04:17:38.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-15T04:17:38.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-09-15T04:17:38.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-09-15T04:17:38.538 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-osd.0.asok 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-09-15T04:17:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.162152/ceph-osd.0.asok log flush 2024-09-15T04:17:38.623 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:17:38.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-09-15T04:17:38.632 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T04:17:38.527+0000 7f080e1fc640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-09-15T04:17:38.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162152/ceph-osd.0.asok 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-09-15T04:17:38.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.162152/ceph-osd.0.asok log reopen 2024-09-15T04:17:38.707 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:17:38.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-15T04:17:38.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:38.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:38.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:38.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:38.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:38.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:38.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:38.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:38.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:38.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:38.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:38.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:38.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:38.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:38.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:38.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:38.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:38.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:38.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:38.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:38.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:38.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:38.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:38.836 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:38.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:38.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:38.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:38.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:38.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:38.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:38.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:38.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:38.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:38.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-09-15T04:17:38.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:17:38.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162152 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-09-15T04:17:38.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-09-15T04:17:38.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-09-15T04:17:38.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-09-15T04:17:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-09-15T04:17:38.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-09-15T04:17:38.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-09-15T04:17:38.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-09-15T04:17:38.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-09-15T04:17:38.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-09-15T04:17:38.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-09-15T04:17:38.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-09-15T04:17:38.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-09-15T04:17:38.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-09-15T04:17:38.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-09-15T04:17:38.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-09-15T04:17:38.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:38.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:38.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:38.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:38.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:38.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:38.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:38.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:38.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:38.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:38.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:38.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:17:38.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-config 0 2024-09-15T04:17:38.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-09-15T04:17:38.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:17:38.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-09-15T04:17:38.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:38.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:38.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:38.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:38.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:38.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:38.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:38.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:38.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:38.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:38.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:38.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:38.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:17:38.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162152 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162152 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:38.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:17:38.948 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:17:38.949 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:17:39.014 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-09-15T04:17:39.014 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-09-15T04:17:39.078 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:17:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:17:39.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-copy-from 2024-09-15T04:17:39.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-09-15T04:17:39.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-09-15T04:17:39.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-09-15T04:17:39.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-09-15T04:17:39.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-09-15T04:17:39.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-09-15T04:17:39.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none ' 2024-09-15T04:17:39.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:17:39.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:17:39.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:17:39.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:17:39.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:17:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:17:39.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:17:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:17:39.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:17:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:17:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166310 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-09-15T04:17:39.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:17:39.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:39.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:39.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.166310 2024-09-15T04:17:39.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-09-15T04:17:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-09-15T04:17:39.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:17:39.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:39.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:39.450 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:39.450 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:39.451 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:39.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:17:39.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:17:39.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:17:39.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:17:39.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.166310/ceph-mon.a.asok 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:17:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.166310/ceph-mon.a.asok config get fsid 2024-09-15T04:17:39.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.166310/ceph-mon.a.asok 2024-09-15T04:17:39.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:17:39.805 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.166310/ceph-mon.a.asok config get mon_host 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-09-15T04:17:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:17:40.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:17:40.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-09-15T04:17:40.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:17:40.732 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:17:40.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=99367192-353f-4e93-908e-2bce998cdd53 2024-09-15T04:17:40.734 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 99367192-353f-4e93-908e-2bce998cdd53 2024-09-15T04:17:40.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 99367192-353f-4e93-908e-2bce998cdd53' 2024-09-15T04:17:40.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:17:40.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkX+ZmITpVLBAAAeffqdtuX37YD2D+JUcrvg== 2024-09-15T04:17:40.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkX+ZmITpVLBAAAeffqdtuX37YD2D+JUcrvg=="}' 2024-09-15T04:17:40.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 99367192-353f-4e93-908e-2bce998cdd53 -i td/osd-copy-from/0/new.json 2024-09-15T04:17:41.050 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:17:41.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-09-15T04:17:41.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkX+ZmITpVLBAAAeffqdtuX37YD2D+JUcrvg== --osd-uuid 99367192-353f-4e93-908e-2bce998cdd53 2024-09-15T04:17:41.082 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.081+0000 7f1c5540f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:41.095 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.094+0000 7f1c5540f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:41.096 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.095+0000 7f1c5540f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.095+0000 7f1c5540f440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-15T04:17:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.095+0000 7f1c5540f440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-15T04:17:41.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.095+0000 7f1c5540f440 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-09-15T04:17:41.135 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:41.134+0000 7f1c5540f440 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-09-15T04:17:46.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-09-15T04:17:46.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:17:46.457 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:17:46.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:17:46.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:17:46.939 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:17:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:17:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:17:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:17:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:17:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:17:46.960 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:46.959+0000 7ff024889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:46.961 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:46.961+0000 7ff024889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:46.962 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:46.962+0000 7ff024889440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:17:47.290 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:17:47.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:47.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:47.479 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:47.479+0000 7ff024889440 -1 Falling back to public interface 2024-09-15T04:17:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:48.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:48.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:48.665 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:17:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:17:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:49.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:49.971 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:17:49.970+0000 7ff024889440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:17:50.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:50.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:50.022 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:17:50.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:17:50.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:50.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:50.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:51.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:51.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:51.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:17:51.565 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:17:51.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:51.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:52.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:53.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:17:53.010 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:17:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:53.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:54.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:54.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:54.399 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:17:54.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:17:54.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:54.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:54.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:55.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:55.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:55.839 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:17:55.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:17:55.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:55.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:56.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:57.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:57.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:57.219 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:17:57.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:17:57.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:57.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:57.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:17:58.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:17:58.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:17:58.693 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:17:58.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:17:58.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:17:58.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:17:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:00.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:00.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:00.048 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:18:00.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:18:00.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:00.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:00.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:01.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:01.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:01.446 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:18:01.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:18:01.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:01.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:02.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:18:03.416 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:18:03.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:03.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:03.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:04.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:04.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:04.981 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:18:04.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:18:04.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:04.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:05.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:06.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:06.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:06.587 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:18:06.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:18:06.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:06.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:07.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:08.382 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:18:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:18:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:08.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:09.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:09.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:09.848 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:18:09.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:18:09.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:09.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:10.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:11.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:11.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:11.613 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:18:11.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:18:11.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:11.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:11.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:12.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:12.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:12.988 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:18:12.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:18:12.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:12.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:13.368 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/167401,v1:127.0.0.1:6803/167401] [v2:127.0.0.1:6804/167401,v1:127.0.0.1:6805/167401] exists,up 99367192-353f-4e93-908e-2bce998cdd53 2024-09-15T04:18:13.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:18:13.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:18:13.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:18:13.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:18:13.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:18:13.375 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:18:13.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b 2024-09-15T04:18:13.377 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b 2024-09-15T04:18:13.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b' 2024-09-15T04:18:13.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:18:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFYOZmvdLhFhAAswoxIlthd1UgkRat5kqyaw== 2024-09-15T04:18:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFYOZmvdLhFhAAswoxIlthd1UgkRat5kqyaw=="}' 2024-09-15T04:18:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b -i td/osd-copy-from/1/new.json 2024-09-15T04:18:13.826 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:18:13.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-09-15T04:18:13.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFYOZmvdLhFhAAswoxIlthd1UgkRat5kqyaw== --osd-uuid 6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b 2024-09-15T04:18:13.860 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.859+0000 7f67e9921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:13.861 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.861+0000 7f67e9921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:13.862 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.862+0000 7f67e9921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:13.863 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.862+0000 7f67e9921440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-15T04:18:13.863 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.862+0000 7f67e9921440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-15T04:18:13.863 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.862+0000 7f67e9921440 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-09-15T04:18:13.891 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:13.890+0000 7f67e9921440 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-09-15T04:18:20.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-09-15T04:18:20.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:18:20.090 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:18:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:18:20.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:18:20.573 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:18:20.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:18:20.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a0e0c14d-459f-4e05-b409-4b2109f4687c --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.166310/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:18:20.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:18:20.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:18:20.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:18:20.594 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:20.593+0000 7f6ceae32440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:20.595 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:20.595+0000 7f6ceae32440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:20.597 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:20.596+0000 7f6ceae32440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:20.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:18:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:18:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:18:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:18:20.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:18:20.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:20.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:18:20.919 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:18:20.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:20.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:21.111 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:21.110+0000 7f6ceae32440 -1 Falling back to public interface 2024-09-15T04:18:21.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:22.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:22.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:22.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:18:22.297 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:18:22.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:22.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:22.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:23.610 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:23.610+0000 7f6ceae32440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:18:23.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:23.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:23.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:18:23.649 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:18:23.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:23.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:24.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:25.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:25.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:25.087 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:18:25.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:18:25.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:25.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:25.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:26.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:26.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:26.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:18:26.517 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:18:26.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:26.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:26.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:27.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:27.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:27.894 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:18:27.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:18:27.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:27.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:28.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:29.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:29.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:29.488 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:18:29.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:18:29.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:29.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:29.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:30.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:30.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:30.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:18:30.895 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:18:30.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:30.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:31.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:32.494 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:18:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:18:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:32.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:33.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:33.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:33.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:18:33.873 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:18:33.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:33.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:34.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:35.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:35.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:35.231 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:18:35.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:18:35.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:35.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:35.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:36.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:36.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:36.635 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:18:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:18:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:37.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:38.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:38.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:38.218 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:18:38.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:18:38.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:38.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:40.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:40.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:40.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:18:40.176 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:18:40.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:40.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:40.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:41.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:41.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:41.716 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:18:41.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:18:41.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:41.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:42.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:43.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:43.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:43.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:18:43.483 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:18:43.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:43.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:44.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:45.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:45.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:45.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:18:45.457 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:18:45.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:45.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:46.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:47.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:47.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:47.301 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:18:47.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:18:47.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:47.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:18:47.718 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/169155,v1:127.0.0.1:6811/169155] [v2:127.0.0.1:6812/169155,v1:127.0.0.1:6813/169155] exists,up 6eab5c90-fbb6-4ba4-ad34-dab2a7771a8b 2024-09-15T04:18:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:18:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:18:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:18:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-09-15T04:18:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T04:18:48.053 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T04:18:48.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T04:18:48.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T04:18:48.782 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T04:18:48.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:18:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T04:18:51.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-09-15T04:18:51.591 INFO:tasks.workunit.client.0.smithi138.stderr:eenvironment:0: which: alias 2024-09-15T04:18:51.591 INFO:tasks.workunit.client.0.smithi138.stderr:eenvironment:1: which: eval declare -f 2024-09-15T04:18:51.591 INFO:tasks.workunit.client.0.smithi138.stderr:eeenvironment:1: which: declare -f 2024-09-15T04:18:51.591 INFO:tasks.workunit.client.0.smithi138.stderr:eenvironment:1: which: /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot rados 2024-09-15T04:18:51.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-09-15T04:18:51.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-09-15T04:18:51.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-09-15T04:18:51.981 INFO:tasks.workunit.client.0.smithi138.stdout:rbd/foo2 mtime 2024-09-15T04:18:51.000000+0000, size 595872 2024-09-15T04:18:51.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-09-15T04:18:52.087 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T04:18:52.112 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T04:18:52.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-15T04:18:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-09-15T04:18:52.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-09-15T04:18:52.202 INFO:tasks.workunit.client.0.smithi138.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-09-15T04:18:52.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-09-15T04:18:52.324 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T04:18:52.353 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T04:18:52.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-09-15T04:18:52.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-09-15T04:18:52.562 INFO:tasks.workunit.client.0.smithi138.stdout:rbd/foo3 mtime 2024-09-15T04:18:51.000000+0000, size 595872 2024-09-15T04:18:52.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-09-15T04:18:52.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-15T04:18:52.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:18:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:18:52.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:18:52.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:18:52.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:18:52.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:18:52.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:18:52.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:18:52.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:18:52.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:18:52.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:18:52.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:18:52.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:18:52.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-15T04:18:52.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:18:52.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:52.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:18:52.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166310 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-copy-from 0 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:18:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:18:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:18:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:18:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:18:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:18:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:18:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:18:52.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:18:52.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:18:52.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:18:52.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:18:52.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:18:52.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-09-15T04:18:52.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:18:52.695 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:52.695 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.166310 2024-09-15T04:18:52.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.166310 2024-09-15T04:18:52.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:18:52.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:18:52.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:18:52.697 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:18:52.697 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:18:52.762 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-09-15T04:18:52.762 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-09-15T04:18:52.824 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:18:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:18:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:18:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:18:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:18:52.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:18:52.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-dup 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-09-15T04:18:52.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none ' 2024-09-15T04:18:52.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-15T04:18:52.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-09-15T04:18:52.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:18:52.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-09-15T04:18:52.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs=TEST_filestore_to_bluestore 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:19: run: for func in $funcs 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:20: run: setup td/osd-dup 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-dup 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-dup 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:18:52.836 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:18:52.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:18:52.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:18:52.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:18:52.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:18:52.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170412 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:18:52.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-dup 2024-09-15T04:18:52.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:18:52.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:52.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.170412 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-dup 1' TERM HUP INT 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:21: run: TEST_filestore_to_bluestore td/osd-dup 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:27: TEST_filestore_to_bluestore: local dir=td/osd-dup 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: ulimit -n 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:29: TEST_filestore_to_bluestore: local flimit=4096 2024-09-15T04:18:52.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:30: TEST_filestore_to_bluestore: '[' 4096 -lt 1536 ']' 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:34: TEST_filestore_to_bluestore: run_mon td/osd-dup a 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-dup 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-dup/a 2024-09-15T04:18:52.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-dup/a --run-dir=td/osd-dup 2024-09-15T04:18:53.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:53.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-dup/a '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --mon-cluster-log-file=td/osd-dup/log --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:18:53.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:18:53.466 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:53.467 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:53.467 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.170412/ceph-mon.a.asok 2024-09-15T04:18:53.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:18:53.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.170412/ceph-mon.a.asok config get fsid 2024-09-15T04:18:53.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:18:53.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.170412/ceph-mon.a.asok 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:18:53.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.170412/ceph-mon.a.asok config get mon_host 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:35: TEST_filestore_to_bluestore: run_mgr td/osd-dup x 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-dup 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-dup/x 2024-09-15T04:18:53.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:18:54.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:18:54.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-dup/x '--log-file=td/osd-dup/$name.log' '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --run-dir=td/osd-dup '--pid-file=td/osd-dup/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:36: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 0 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/0 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/0' 2024-09-15T04:18:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T04:18:54.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/0 2024-09-15T04:18:54.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T04:18:54.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:18:54.482 INFO:tasks.workunit.client.0.smithi138.stdout:add osd 4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:18:54.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd 4b689482-b2d4-475d-b803-08e6e33ea414' 2024-09-15T04:18:54.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T04:18:54.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAuYOZmz0dfHRAAd5scplcd0Yd7QMJ+8/w54w== 2024-09-15T04:18:54.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAuYOZmz0dfHRAAd5scplcd0Yd7QMJ+8/w54w=="}' 2024-09-15T04:18:54.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 4b689482-b2d4-475d-b803-08e6e33ea414 -i td/osd-dup/0/new.json 2024-09-15T04:18:54.803 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:18:54.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/0/new.json 2024-09-15T04:18:54.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAuYOZmz0dfHRAAd5scplcd0Yd7QMJ+8/w54w== --osd-uuid 4b689482-b2d4-475d-b803-08e6e33ea414 --osd-objectstore=filestore 2024-09-15T04:18:54.837 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:54.837+0000 7f5123528440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:54.849 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:54.849+0000 7f5123528440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:54.850 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:54.850+0000 7f5123528440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:55.292 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:55.291+0000 7f5123528440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:18:55.786 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:55.786+0000 7f5123528440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:18:55.787 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:55.786+0000 7f5123528440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:18:55.787 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:55.786+0000 7f5123528440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:18:56.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/0/keyring 2024-09-15T04:18:56.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T04:18:56.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-15T04:18:56.351 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:18:56.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:18:56.492 INFO:tasks.workunit.client.0.smithi138.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-15T04:18:56.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-15T04:18:56.495 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:18:56.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:18:56.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T04:18:56.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T04:18:56.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T04:18:56.518 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:56.517+0000 7f390b209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:56.519 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:56.519+0000 7f390b209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:56.521 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:56.520+0000 7f390b209440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:56.522 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:56.522+0000 7f390b209440 -1 Falling back to public interface 2024-09-15T04:18:56.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-15T04:18:56.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:18:56.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:18:56.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:56.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:57.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:18:57.077 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:57.076+0000 7f390b209440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:18:57.084 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:57.084+0000 7f390b209440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:18:57.084 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:57.084+0000 7f390b209440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:18:57.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:57.088+0000 7f390b209440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:18:58.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:18:58.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:18:58.045 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:18:58.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:18:58.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:18:58.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:18:58.309 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/170714,v1:127.0.0.1:6801/170714] [v2:127.0.0.1:6802/170714,v1:127.0.0.1:6803/170714] exists,up 4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:18:58.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:18:58.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:18:58.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:18:58.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: cat td/osd-dup/osd.0.pid 2024-09-15T04:18:58.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:37: TEST_filestore_to_bluestore: osd_pid=170714 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:38: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 1 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/1 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:18:58.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/1' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/1/journal' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T04:18:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/1 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=184b653d-0dcb-4ec2-84a8-5c48ce711918 2024-09-15T04:18:58.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd 184b653d-0dcb-4ec2-84a8-5c48ce711918' 2024-09-15T04:18:58.317 INFO:tasks.workunit.client.0.smithi138.stdout:add osd 184b653d-0dcb-4ec2-84a8-5c48ce711918 2024-09-15T04:18:58.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T04:18:58.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQAyYOZmdEuPExAA5SCPI8sJ5K5KSgI4LoiDNA== 2024-09-15T04:18:58.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQAyYOZmdEuPExAA5SCPI8sJ5K5KSgI4LoiDNA=="}' 2024-09-15T04:18:58.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 184b653d-0dcb-4ec2-84a8-5c48ce711918 -i td/osd-dup/1/new.json 2024-09-15T04:18:58.803 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:18:58.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/1/new.json 2024-09-15T04:18:58.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAyYOZmdEuPExAA5SCPI8sJ5K5KSgI4LoiDNA== --osd-uuid 184b653d-0dcb-4ec2-84a8-5c48ce711918 --osd-objectstore=filestore 2024-09-15T04:18:58.839 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:58.838+0000 7f5f77760440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:58.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:58.840+0000 7f5f77760440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:58.841 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:58.841+0000 7f5f77760440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:18:59.280 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:59.280+0000 7f5f77760440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:18:59.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:59.766+0000 7f5f77760440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:18:59.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:59.766+0000 7f5f77760440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:18:59.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:18:59.766+0000 7f5f77760440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:19:00.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/1/keyring 2024-09-15T04:19:00.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T04:19:00.499 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:19:00.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-15T04:19:00.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:19:00.857 INFO:tasks.workunit.client.0.smithi138.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-15T04:19:00.860 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:19:00.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-15T04:19:00.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/1 --osd-journal=td/osd-dup/1/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:19:00.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T04:19:00.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T04:19:00.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T04:19:00.882 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:00.881+0000 7fecb0721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:00.883 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:00.883+0000 7fecb0721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:00.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:00.884+0000 7fecb0721440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:00.886 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:00.885+0000 7fecb0721440 -1 Falling back to public interface 2024-09-15T04:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-15T04:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:19:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:01.267 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:19:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:19:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:01.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:19:01.299 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:01.299+0000 7fecb0721440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:19:01.307 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:01.307+0000 7fecb0721440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:19:01.307 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:01.307+0000 7fecb0721440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:19:01.311 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:01.311+0000 7fecb0721440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:19:01.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:19:02.646 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:19:02.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:02.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:19:03.000 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/171076,v1:127.0.0.1:6811/171076] [v2:127.0.0.1:6812/171076,v1:127.0.0.1:6813/171076] exists,up 184b653d-0dcb-4ec2-84a8-5c48ce711918 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:39: TEST_filestore_to_bluestore: run_osd_filestore td/osd-dup 2 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-dup 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-15T04:19:03.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-dup/2 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-dup/2' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-dup/2/journal' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-dup' 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T04:19:03.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-15T04:19:03.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-dup/2 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=1f976bc1-1200-42a3-bdc5-212933764414 2024-09-15T04:19:03.007 INFO:tasks.workunit.client.0.smithi138.stdout:add osd 1f976bc1-1200-42a3-bdc5-212933764414 2024-09-15T04:19:03.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd 1f976bc1-1200-42a3-bdc5-212933764414' 2024-09-15T04:19:03.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T04:19:03.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQA3YOZm3sUJARAAouLUI0kW9axgkzieZ/Q98w== 2024-09-15T04:19:03.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQA3YOZm3sUJARAAouLUI0kW9axgkzieZ/Q98w=="}' 2024-09-15T04:19:03.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 1f976bc1-1200-42a3-bdc5-212933764414 -i td/osd-dup/2/new.json 2024-09-15T04:19:03.460 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:19:03.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-dup/2/new.json 2024-09-15T04:19:03.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA3YOZm3sUJARAAouLUI0kW9axgkzieZ/Q98w== --osd-uuid 1f976bc1-1200-42a3-bdc5-212933764414 --osd-objectstore=filestore 2024-09-15T04:19:03.496 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:03.495+0000 7f6f28d4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:03.497 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:03.497+0000 7f6f28d4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:03.498 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:03.498+0000 7f6f28d4a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:03.919 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:03.919+0000 7f6f28d4a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:19:04.558 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:04.557+0000 7f6f28d4a440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:19:04.565 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:04.565+0000 7f6f28d4a440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:19:04.565 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:04.565+0000 7f6f28d4a440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T04:19:05.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-dup/2/keyring 2024-09-15T04:19:05.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T04:19:05.228 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:19:05.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-15T04:19:05.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-dup/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:19:05.545 INFO:tasks.workunit.client.0.smithi138.stderr:Error EINVAL: key for osd.2 not found in provided keyring 2024-09-15T04:19:05.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-15T04:19:05.549 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:19:05.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/2 --osd-journal=td/osd-dup/2/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:19:05.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T04:19:05.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T04:19:05.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T04:19:05.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:05.570+0000 7ff7bef23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:05.573 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:05.572+0000 7ff7bef23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:05.574 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:05.573+0000 7ff7bef23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:05.575 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:05.575+0000 7ff7bef23440 -1 Falling back to public interface 2024-09-15T04:19:05.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:05.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:19:05.961 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:19:05.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:05.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:19:06.082 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:06.082+0000 7ff7bef23440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T04:19:06.098 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:06.098+0000 7ff7bef23440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:19:06.099 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:06.098+0000 7ff7bef23440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T04:19:06.103 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:06.103+0000 7ff7bef23440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:19:06.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:07.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:07.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:07.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:19:07.455 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:19:07.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:07.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:19:07.856 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/171359,v1:127.0.0.1:6819/171359] [v2:127.0.0.1:6820/171359,v1:127.0.0.1:6821/171359] exists,up 1f976bc1-1200-42a3-bdc5-212933764414 2024-09-15T04:19:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:19:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:19:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:19:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:41: TEST_filestore_to_bluestore: sleep 5 2024-09-15T04:19:12.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:43: TEST_filestore_to_bluestore: create_pool foo 16 2024-09-15T04:19:12.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-15T04:19:13.338 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T04:19:13.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:19:14.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:46: TEST_filestore_to_bluestore: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-15T04:19:14.419 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:19:14.419 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-15T04:19:14.419 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_171539 2024-09-15T04:19:14.419 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:19:14.419 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:19:15.419 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 16 0 0 0 - 0 2024-09-15T04:19:16.419 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 16 0 0 0 - 0 2024-09-15T04:19:17.419 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 17 1 0.00130193 0.00130208 2.79899 2.79899 2024-09-15T04:19:18.419 INFO:tasks.workunit.client.0.smithi138.stdout: 4 16 25 9 0.00878796 0.03125 3.38051 3.29843 2024-09-15T04:19:19.419 INFO:tasks.workunit.client.0.smithi138.stdout: 5 16 25 9 0.00703031 0 - 3.29843 2024-09-15T04:19:20.420 INFO:tasks.workunit.client.0.smithi138.stdout: 6 16 101 85 0.0553315 0.148438 0.114693 1.1078 2024-09-15T04:19:21.420 INFO:tasks.workunit.client.0.smithi138.stdout: 7 16 229 213 0.118846 0.5 0.120663 0.516218 2024-09-15T04:19:22.420 INFO:tasks.workunit.client.0.smithi138.stdout: 8 16 325 309 0.150859 0.375 0.169663 0.396342 2024-09-15T04:19:23.420 INFO:tasks.workunit.client.0.smithi138.stdout: 9 16 405 389 0.168815 0.3125 0.174092 0.364796 2024-09-15T04:19:24.420 INFO:tasks.workunit.client.0.smithi138.stdout: 10 16 501 485 0.189428 0.375 0.119947 0.326739 2024-09-15T04:19:25.420 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 10.0758 2024-09-15T04:19:25.420 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 501 2024-09-15T04:19:25.420 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.194231 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.196622 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.5 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 49 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 50.368 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 128 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 0 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.320119 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.776884 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 5.48845 2024-09-15T04:19:25.421 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.102999 2024-09-15T04:19:25.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 170714 2024-09-15T04:19:25.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: sleep 1 2024-09-15T04:19:26.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:49: TEST_filestore_to_bluestore: kill 170714 2024-09-15T04:19:26.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh: line 49: kill: (170714) - No such process 2024-09-15T04:19:26.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:50: TEST_filestore_to_bluestore: ceph osd down 0 2024-09-15T04:19:26.881 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T04:19:26.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:52: TEST_filestore_to_bluestore: mv td/osd-dup/0 td/osd-dup/0.old 2024-09-15T04:19:26.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:53: TEST_filestore_to_bluestore: mkdir td/osd-dup/0 2024-09-15T04:19:26.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: cat td/osd-dup/0.old/fsid 2024-09-15T04:19:26.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:54: TEST_filestore_to_bluestore: ofsid=4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:19:26.895 INFO:tasks.workunit.client.0.smithi138.stdout:osd fsid 4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:19:26.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:55: TEST_filestore_to_bluestore: echo 'osd fsid 4b689482-b2d4-475d-b803-08e6e33ea414' 2024-09-15T04:19:26.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:56: TEST_filestore_to_bluestore: O='--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:19:26.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:57: TEST_filestore_to_bluestore: CEPH_ARGS+='--log-file td/osd-dup/cot.log --log-max-recent 0 ' 2024-09-15T04:19:26.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:58: TEST_filestore_to_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-dup/0 --fsid 4b689482-b2d4-475d-b803-08e6e33ea414 --op mkfs --no-mon-config 2024-09-15T04:19:31.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:60: TEST_filestore_to_bluestore: ceph-objectstore-tool --data-path td/osd-dup/0.old --target-data-path td/osd-dup/0 --op dup 2024-09-15T04:19:31.341 INFO:tasks.workunit.client.0.smithi138.stdout:dup from filestore: td/osd-dup/0.old 2024-09-15T04:19:31.341 INFO:tasks.workunit.client.0.smithi138.stdout: to bluestore: td/osd-dup/0 2024-09-15T04:19:32.459 INFO:tasks.workunit.client.0.smithi138.stdout:fsid 4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:19:32.459 INFO:tasks.workunit.client.0.smithi138.stdout:17 collections 2024-09-15T04:19:32.459 INFO:tasks.workunit.client.0.smithi138.stdout:1/17 1.3_head 2024-09-15T04:19:32.793 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2024-09-15T04:19:32.793 INFO:tasks.workunit.client.0.smithi138.stdout:2/17 1.1_head 2024-09-15T04:19:32.987 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 32 objects, 126976 bytes, 37 keys 2024-09-15T04:19:32.988 INFO:tasks.workunit.client.0.smithi138.stdout:3/17 1.8_head 2024-09-15T04:19:33.174 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2024-09-15T04:19:33.175 INFO:tasks.workunit.client.0.smithi138.stdout:4/17 1.d_head 2024-09-15T04:19:33.370 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 35 objects, 135192 bytes, 40 keys 2024-09-15T04:19:33.370 INFO:tasks.workunit.client.0.smithi138.stdout:5/17 1.f_head 2024-09-15T04:19:33.510 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 26 objects, 102400 bytes, 31 keys 2024-09-15T04:19:33.510 INFO:tasks.workunit.client.0.smithi138.stdout:6/17 1.7_head 2024-09-15T04:19:33.721 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 35 objects, 139264 bytes, 40 keys 2024-09-15T04:19:33.721 INFO:tasks.workunit.client.0.smithi138.stdout:7/17 1.5_head 2024-09-15T04:19:33.925 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2024-09-15T04:19:33.925 INFO:tasks.workunit.client.0.smithi138.stdout:8/17 1.e_head 2024-09-15T04:19:34.145 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2024-09-15T04:19:34.146 INFO:tasks.workunit.client.0.smithi138.stdout:9/17 1.6_head 2024-09-15T04:19:34.293 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2024-09-15T04:19:34.293 INFO:tasks.workunit.client.0.smithi138.stdout:10/17 meta 2024-09-15T04:19:34.296 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 46 objects, 47132 bytes, 1 keys 2024-09-15T04:19:34.296 INFO:tasks.workunit.client.0.smithi138.stdout:11/17 1.a_head 2024-09-15T04:19:34.404 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 32 objects, 126976 bytes, 37 keys 2024-09-15T04:19:34.405 INFO:tasks.workunit.client.0.smithi138.stdout:12/17 1.c_head 2024-09-15T04:19:34.564 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 35 objects, 139264 bytes, 40 keys 2024-09-15T04:19:34.564 INFO:tasks.workunit.client.0.smithi138.stdout:13/17 1.9_head 2024-09-15T04:19:34.748 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 38 objects, 151552 bytes, 43 keys 2024-09-15T04:19:34.749 INFO:tasks.workunit.client.0.smithi138.stdout:14/17 1.0_head 2024-09-15T04:19:34.910 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 33 objects, 131072 bytes, 38 keys 2024-09-15T04:19:34.910 INFO:tasks.workunit.client.0.smithi138.stdout:15/17 1.b_head 2024-09-15T04:19:35.070 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 34 objects, 135168 bytes, 39 keys 2024-09-15T04:19:35.070 INFO:tasks.workunit.client.0.smithi138.stdout:16/17 1.4_head 2024-09-15T04:19:35.187 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 28 objects, 110592 bytes, 33 keys 2024-09-15T04:19:35.188 INFO:tasks.workunit.client.0.smithi138.stdout:17/17 1.2_head 2024-09-15T04:19:35.190 INFO:tasks.workunit.client.0.smithi138.stdout: 0 objects, 0 bytes, 0 keys 29 objects, 114688 bytes, 34 keys 2024-09-15T04:19:35.190 INFO:tasks.workunit.client.0.smithi138.stdout:keyring 2024-09-15T04:19:35.190 INFO:tasks.workunit.client.0.smithi138.stdout:duping osd metadata 2024-09-15T04:19:35.581 INFO:tasks.workunit.client.0.smithi138.stdout:done. 2024-09-15T04:19:36.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:62: TEST_filestore_to_bluestore: CEPH_ARGS='--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:19:36.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:64: TEST_filestore_to_bluestore: activate_osd td/osd-dup 0 2024-09-15T04:19:36.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-dup 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-dup/0 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-dup/0' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-dup/0/journal' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:19:36.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-dup' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-dup/$name.log' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-dup/$name.pid' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:19:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-dup/0 2024-09-15T04:19:36.400 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:19:36.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5c94d11f-7358-48e4-b896-dc79579bd201 --auth-supported=none --mon-host=127.0.0.1:7146 --filestore_wbthrottle_xfs_ios_hard_limit=900 --filestore_wbthrottle_btrfs_ios_hard_limit=900 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-dup/0 --osd-journal=td/osd-dup/0/journal --chdir= --run-dir=td/osd-dup '--admin-socket=/tmp/ceph-asok.170412/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-dup/$name.log' '--pid-file=td/osd-dup/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-dup/0/whoami 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:19:36.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:19:36.422 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:36.422+0000 7fb3bf075440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:36.425 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:36.424+0000 7fb3bf075440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:36.426 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:36.425+0000 7fb3bf075440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:19:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:19:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:19:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:19:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:19:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:19:36.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:36.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:19:36.799 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:19:36.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:36.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:36.942 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:36.942+0000 7fb3bf075440 -1 Falling back to public interface 2024-09-15T04:19:37.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:38.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:38.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:38.200 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:19:38.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:19:38.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:38.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:38.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:39.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:39.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:39.558 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:19:39.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:19:39.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:39.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:39.734 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:19:39.733+0000 7fb3bf075440 -1 osd.0 21 log_to_monitors true 2024-09-15T04:19:39.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:40.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:40.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:40.946 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:19:40.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:19:40.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:40.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:41.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:19:42.322 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:19:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:42.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:42.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:43.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:43.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:43.823 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:19:43.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:19:43.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:43.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:44.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:19:45.226 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:19:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:45.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:45.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:46.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:46.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:46.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:19:46.590 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:19:46.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:46.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:47.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:48.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:48.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:48.080 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:19:48.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:19:48.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:48.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:48.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:49.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:49.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:49.477 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:19:49.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:19:49.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:49.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:49.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:50.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:50.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:50.863 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:19:50.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:19:50.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:50.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:52.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:52.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:52.513 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:19:52.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:19:52.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:52.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:53.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:54.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:54.312 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:19:54.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:19:54.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:55.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:56.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:56.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:56.096 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:19:56.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:19:56.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:56.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:57.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:58.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:58.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:58.345 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:19:58.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:19:58.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:58.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:19:58.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:19:59.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:19:59.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:19:59.872 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:19:59.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:19:59.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:19:59.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:00.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:01.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:01.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:01.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:20:01.328 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:20:01.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:01.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:01.702 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 25 up_thru 25 down_at 22 last_clean_interval [5,21) [v2:127.0.0.1:6800/172412,v1:127.0.0.1:6801/172412] [v2:127.0.0.1:6802/172412,v1:127.0.0.1:6803/172412] exists,up 4b689482-b2d4-475d-b803-08e6e33ea414 2024-09-15T04:20:01.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:20:01.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:20:01.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:20:01.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: ceph osd stat 2024-09-15T04:20:01.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:66: TEST_filestore_to_bluestore: grep '3 up' 2024-09-15T04:20:02.107 INFO:tasks.workunit.client.0.smithi138.stdout:3 osds: 3 up (since 1.90027s), 3 in (since 58s); epoch: e27 2024-09-15T04:20:02.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: ceph osd metadata 0 2024-09-15T04:20:02.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:67: TEST_filestore_to_bluestore: grep bluestore 2024-09-15T04:20:02.462 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/osd-dup/0/block", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T04:20:02.463 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T04:20:02.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:69: TEST_filestore_to_bluestore: ceph osd scrub 0 2024-09-15T04:20:02.795 INFO:tasks.workunit.client.0.smithi138.stderr:instructed osd(s) 0 to scrub 2024-09-15T04:20:02.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:72: TEST_filestore_to_bluestore: sleep 15 2024-09-15T04:20:17.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:74: TEST_filestore_to_bluestore: flush_pg_stats 2024-09-15T04:20:17.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:20:17.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:18.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:20:18.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182410 2024-09-15T04:20:18.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182410 2024-09-15T04:20:18.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410' 2024-09-15T04:20:18.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:18.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:20:18.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-15T04:20:18.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-15T04:20:18.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410 1-42949672978' 2024-09-15T04:20:18.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:18.485 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:20:18.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509458 2024-09-15T04:20:18.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509458 2024-09-15T04:20:18.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182410 1-42949672978 2-64424509458' 2024-09-15T04:20:18.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:18.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182410 2024-09-15T04:20:18.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:18.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:20:18.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182410 2024-09-15T04:20:18.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:18.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182410 2024-09-15T04:20:18.618 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 107374182410 2024-09-15T04:20:18.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182410' 2024-09-15T04:20:18.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:20:18.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182408 -lt 107374182410 2024-09-15T04:20:18.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:20:19.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:20:19.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:20:20.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182411 -lt 107374182410 2024-09-15T04:20:20.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:20.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-15T04:20:20.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:20.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:20:20.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-15T04:20:20.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:20.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-15T04:20:20.330 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672978 2024-09-15T04:20:20.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-15T04:20:20.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:20:20.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-15T04:20:20.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:20.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509458 2024-09-15T04:20:20.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:20.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:20:20.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509458 2024-09-15T04:20:20.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:20.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509458 2024-09-15T04:20:20.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-09-15T04:20:20.686 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509458 2024-09-15T04:20:20.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:20:21.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509458 -lt 64424509458 2024-09-15T04:20:21.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:76: TEST_filestore_to_bluestore: wait_for_clean 2024-09-15T04:20:21.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:20:21.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:20:21.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:20:21.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:20:21.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:20:21.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:20:21.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:20:21.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:20:21.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:20:21.502 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:20:21.502 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T04:20:21.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:20:21.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:21.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:20:21.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-15T04:20:21.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-15T04:20:21.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413' 2024-09-15T04:20:21.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:21.607 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:20:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-15T04:20:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-15T04:20:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413 1-42949672980' 2024-09-15T04:20:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:20:21.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:20:21.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509460 2024-09-15T04:20:21.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509460 2024-09-15T04:20:21.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-107374182413 1-42949672980 2-64424509460' 2024-09-15T04:20:21.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:21.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-107374182413 2024-09-15T04:20:21.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:21.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:20:21.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-107374182413 2024-09-15T04:20:21.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:21.820 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 107374182413 2024-09-15T04:20:21.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-15T04:20:21.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 107374182413' 2024-09-15T04:20:21.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:20:22.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182411 -lt 107374182413 2024-09-15T04:20:22.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:20:23.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:20:23.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:20:23.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182413 -lt 107374182413 2024-09-15T04:20:23.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:23.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-15T04:20:23.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:23.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:20:23.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-15T04:20:23.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:23.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-15T04:20:23.526 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672980 2024-09-15T04:20:23.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T04:20:23.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:20:23.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-15T04:20:23.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:20:23.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509460 2024-09-15T04:20:23.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:20:23.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:20:23.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509460 2024-09-15T04:20:23.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:20:23.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509460 2024-09-15T04:20:23.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-15T04:20:23.894 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509460 2024-09-15T04:20:23.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:20:24.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509461 -lt 64424509460 2024-09-15T04:20:24.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:20:24.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:20:24.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:20:24.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:20:25.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-09-15T04:20:25.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:20:25.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:20:25.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:22: run: teardown td/osd-dup 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-15T04:20:25.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:20:25.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:20:25.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:20:25.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:20:25.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:20:25.721 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:20:25.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:20:25.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:20:25.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:20:25.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:25.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:20:25.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:20:25.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:25.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:20:25.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:20:25.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-15T04:20:25.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:20:25.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:25.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:20:25.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170412 2024-09-15T04:20:25.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:20:25.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:20:25.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-dup 0 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:20:25.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:20:25.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:20:25.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:20:25.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:20:25.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:20:25.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:20:25.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:20:25.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:20:25.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:25.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:20:25.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:20:25.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:25.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:20:25.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:20:25.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-09-15T04:20:25.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:20:25.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:25.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.170412 2024-09-15T04:20:25.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.170412 2024-09-15T04:20:25.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:20:25.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:20:25.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:20:25.775 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:20:25.775 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:20:25.839 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-09-15T04:20:25.974 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-09-15T04:20:25.996 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:20:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-fast-mark-down 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-09-15T04:20:26.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-09-15T04:20:26.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none ' 2024-09-15T04:20:26.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none ' 2024-09-15T04:20:26.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-09-15T04:20:26.005 INFO:tasks.workunit.client.0.smithi138.stdout:Ensuring old behavior is there... 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:20:26.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:20:26.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:26.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:20:26.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:20:26.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-15T04:20:26.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:20:26.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173995 2024-09-15T04:20:26.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:20:26.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:20:26.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-15T04:20:26.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:20:26.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173995 2024-09-15T04:20:26.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:20:26.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:20:26.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:20:26.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:20:26.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-15T04:20:26.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-15T04:20:26.033 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.032+0000 7f625ba55900 -1 failed for service _ceph-mon._tcp 2024-09-15T04:20:26.033 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.032+0000 7f625ba55900 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T04:20:26.033 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T04:20:26.033 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.032+0000 7f625ba55900 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:20:26.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T04:20:26.652 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.652+0000 7f912d80a900 -1 failed for service _ceph-mon._tcp 2024-09-15T04:20:26.653 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.652+0000 7f912d80a900 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T04:20:26.653 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T04:20:26.653 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:26.652+0000 7f912d80a900 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-15T04:20:26.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-09-15T04:20:26.657 INFO:tasks.workunit.client.0.smithi138.stdout:Testing async msgr... 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:20:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:20:26.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:20:26.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:20:26.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:20:26.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:20:26.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:20:26.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:20:26.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:26.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:20:26.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:20:26.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:20:26.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:20:26.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:20:26.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-15T04:20:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:20:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173995 2024-09-15T04:20:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:20:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:20:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-09-15T04:20:26.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:20:26.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.173995 2024-09-15T04:20:26.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:20:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:20:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:20:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-09-15T04:20:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-09-15T04:20:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-09-15T04:20:26.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:20:26.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:20:26.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:20:26.962 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:20:26.962 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:26.962 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:26.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:20:26.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T04:20:27.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:20:27.212 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:27.213 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:27.213 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173995/ceph-mon.a.asok 2024-09-15T04:20:27.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:20:27.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173995/ceph-mon.a.asok config get fsid 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:20:27.290 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.173995/ceph-mon.a.asok 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:20:27.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.173995/ceph-mon.a.asok config get mon_host 2024-09-15T04:20:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-09-15T04:20:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-09-15T04:20:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:20:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:20:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:20:27.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-09-15T04:20:27.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:28.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:20:28.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:20:28.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:20:28.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:20:28.227 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:20:28.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dcf792e0-3160-4935-8d52-571a61d2bf4b 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 dcf792e0-3160-4935-8d52-571a61d2bf4b' 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 dcf792e0-3160-4935-8d52-571a61d2bf4b 2024-09-15T04:20:28.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:20:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCMYOZmv6FdDhAAt6/j4iRje0CpFI9Ba0CVNw== 2024-09-15T04:20:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCMYOZmv6FdDhAAt6/j4iRje0CpFI9Ba0CVNw=="}' 2024-09-15T04:20:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dcf792e0-3160-4935-8d52-571a61d2bf4b -i td/osd-fast-mark-down/0/new.json 2024-09-15T04:20:28.549 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:20:28.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-09-15T04:20:28.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCMYOZmv6FdDhAAt6/j4iRje0CpFI9Ba0CVNw== --osd-uuid dcf792e0-3160-4935-8d52-571a61d2bf4b 2024-09-15T04:20:28.585 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.585+0000 7f0623733440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.595+0000 7f0623733440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:28.597 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.597+0000 7f0623733440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:28.598 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.597+0000 7f0623733440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-15T04:20:28.598 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.597+0000 7f0623733440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-15T04:20:28.598 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.597+0000 7f0623733440 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-09-15T04:20:28.636 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:28.635+0000 7f0623733440 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-09-15T04:20:33.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-09-15T04:20:33.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:20:33.697 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:20:33.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:20:33.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:20:34.179 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:20:34.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:20:34.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:20:34.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:20:34.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:20:34.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:20:34.204 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:34.203+0000 7f6a3604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:34.222 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:34.222+0000 7f6a3604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:34.223 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:34.223+0000 7f6a3604f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:20:34.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:34.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:20:34.603 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:20:34.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:34.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:34.756 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:34.756+0000 7f6a3604f440 -1 Falling back to public interface 2024-09-15T04:20:34.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:35.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:35.996 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:20:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:20:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:36.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:37.276 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:20:37.275+0000 7f6a3604f440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:20:37.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:37.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:37.359 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:20:37.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:20:37.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:37.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:37.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:38.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:38.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:38.864 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:20:38.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:20:38.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:38.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:40.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:40.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:40.234 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:20:40.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:20:40.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:40.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:40.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:41.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:41.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:41.615 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:20:41.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:20:41.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:41.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:42.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:43.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:43.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:43.040 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:20:43.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:20:43.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:43.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:43.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:44.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:44.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:44.430 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:20:44.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:20:44.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:44.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:44.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:45.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:45.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:45.889 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:20:45.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:20:45.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:45.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:46.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:47.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:47.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:47.251 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:20:47.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:20:47.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:47.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:48.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:48.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:48.665 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:20:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:20:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:48.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:49.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:50.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:50.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:50.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:20:50.402 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:20:50.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:50.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:51.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:52.110 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:20:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:20:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:52.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:53.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:53.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:53.890 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:20:53.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:20:53.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:53.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:54.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:55.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:55.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:55.768 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:20:55.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:20:55.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:55.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:57.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:57.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:57.376 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:20:57.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:20:57.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:57.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:58.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:20:59.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:20:59.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:20:59.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:20:59.014 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:20:59.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:20:59.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:20:59.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:00.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:00.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:00.977 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:21:00.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:21:00.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:00.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:21:01.360 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/175151,v1:127.0.0.1:6803/175151] [v2:127.0.0.1:6804/175151,v1:127.0.0.1:6805/175151] exists,up dcf792e0-3160-4935-8d52-571a61d2bf4b 2024-09-15T04:21:01.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:21:01.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:21:01.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:21:01.361 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=175151 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-09-15T04:21:01.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:21:01.366 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:21:01.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2883be10-23fe-4a90-ba66-dc6dfbfc8ff6 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 2883be10-23fe-4a90-ba66-dc6dfbfc8ff6 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2883be10-23fe-4a90-ba66-dc6dfbfc8ff6' 2024-09-15T04:21:01.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:21:01.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtYOZmsBmhFhAAp3W5EZrK3fHJwY5JhEKtxA== 2024-09-15T04:21:01.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtYOZmsBmhFhAAp3W5EZrK3fHJwY5JhEKtxA=="}' 2024-09-15T04:21:01.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2883be10-23fe-4a90-ba66-dc6dfbfc8ff6 -i td/osd-fast-mark-down/1/new.json 2024-09-15T04:21:01.869 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:21:01.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-09-15T04:21:01.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtYOZmsBmhFhAAp3W5EZrK3fHJwY5JhEKtxA== --osd-uuid 2883be10-23fe-4a90-ba66-dc6dfbfc8ff6 2024-09-15T04:21:01.906 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.906+0000 7f894be5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:01.928 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.927+0000 7f894be5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:01.929 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.928+0000 7f894be5a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:01.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.929+0000 7f894be5a440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-15T04:21:01.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.929+0000 7f894be5a440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-15T04:21:01.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:01.929+0000 7f894be5a440 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-09-15T04:21:02.012 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:02.011+0000 7f894be5a440 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-09-15T04:21:07.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-09-15T04:21:07.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:21:07.765 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:21:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:21:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:21:08.242 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:21:08.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:21:08.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:21:08.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:21:08.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:21:08.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:21:08.266 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:08.266+0000 7ff26011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:08.268 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:08.267+0000 7ff26011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:08.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:08.268+0000 7ff26011f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:08.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:21:08.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:21:08.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:21:08.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:08.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:08.786 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:08.786+0000 7ff26011f440 -1 Falling back to public interface 2024-09-15T04:21:08.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:09.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:09.956 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:21:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:21:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:09.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:10.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:11.261+0000 7ff26011f440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:21:11.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:11.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:11.319 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:21:11.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:21:11.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:11.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:11.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:12.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:12.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:12.973 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:21:12.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:21:12.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:12.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:13.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:14.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:14.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:14.346 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:21:14.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:21:14.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:14.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:14.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:15.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:15.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:15.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:21:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:21:15.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:15.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:16.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:17.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:17.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:17.245 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:21:17.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:21:17.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:17.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:17.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:18.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:18.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:18.619 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:21:18.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:21:18.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:18.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:19.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:20.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:20.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:20.020 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:21:20.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:21:20.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:20.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:20.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:21.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:21.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:21.386 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:21:21.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:21:21.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:21.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:21.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:22.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:22.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:22.782 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:21:22.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:21:22.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:22.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:23.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:24.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:24.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:24.390 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:21:24.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:21:24.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:24.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:25.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:26.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:26.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:26.284 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:21:26.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:21:26.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:26.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:26.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:27.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:27.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:27.955 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:21:27.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:21:27.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:27.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:28.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:29.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:29.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:29.538 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:21:29.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:21:29.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:29.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:30.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:31.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:31.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:31.259 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:21:31.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:21:31.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:31.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:32.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:33.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:33.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:33.169 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:21:33.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:21:33.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:33.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:33.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:34.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:34.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:34.978 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:21:34.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:21:34.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:34.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:21:35.370 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/176906,v1:127.0.0.1:6811/176906] [v2:127.0.0.1:6812/176906,v1:127.0.0.1:6813/176906] exists,up 2883be10-23fe-4a90-ba66-dc6dfbfc8ff6 2024-09-15T04:21:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:21:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:21:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:21:35.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-09-15T04:21:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=176906 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:21:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:21:35.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:21:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7add4a90-0765-425d-82a3-2d00ccf5a6ea 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7add4a90-0765-425d-82a3-2d00ccf5a6ea' 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 7add4a90-0765-425d-82a3-2d00ccf5a6ea 2024-09-15T04:21:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:21:35.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPYOZm6aVNFxAA5X+j/1XA2IXlGh3BXLTTkw== 2024-09-15T04:21:35.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPYOZm6aVNFxAA5X+j/1XA2IXlGh3BXLTTkw=="}' 2024-09-15T04:21:35.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7add4a90-0765-425d-82a3-2d00ccf5a6ea -i td/osd-fast-mark-down/2/new.json 2024-09-15T04:21:35.936 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:21:35.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-09-15T04:21:35.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPYOZm6aVNFxAA5X+j/1XA2IXlGh3BXLTTkw== --osd-uuid 7add4a90-0765-425d-82a3-2d00ccf5a6ea 2024-09-15T04:21:35.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:35.971+0000 7f135d60b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:36.005 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.003+0000 7f135d60b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:36.005 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.004+0000 7f135d60b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:36.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.004+0000 7f135d60b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-15T04:21:36.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.004+0000 7f135d60b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-15T04:21:36.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.004+0000 7f135d60b440 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-09-15T04:21:36.045 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:36.044+0000 7f135d60b440 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-09-15T04:21:41.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-09-15T04:21:41.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:21:41.877 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:21:41.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:21:41.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:21:42.359 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:21:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:21:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=6e9fea43-28c3-4bae-a4c4-3c98afed1bfc --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.173995/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:21:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:21:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:21:42.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:21:42.383 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:42.382+0000 7f4e7ed57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:42.385 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:42.384+0000 7f4e7ed57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:42.386 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:42.385+0000 7f4e7ed57440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:21:42.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:21:42.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:21:42.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:21:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:21:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:21:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:21:42.709 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:21:42.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:42.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:42.902 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:42.902+0000 7f4e7ed57440 -1 Falling back to public interface 2024-09-15T04:21:43.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:44.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:44.060 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:21:44.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:44.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:21:44.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:44.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:44.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:45.399 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:21:45.398+0000 7f4e7ed57440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:21:45.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:45.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:45.415 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:21:45.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:21:45.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:45.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:45.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:46.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:46.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:46.901 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:21:46.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:21:46.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:46.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:47.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:48.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:48.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:48.324 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:21:48.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:21:48.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:48.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:48.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:49.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:49.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:49.747 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:21:49.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:21:49.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:49.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:50.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:51.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:51.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:51.221 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:21:51.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:21:51.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:51.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:51.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:52.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:52.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:52.618 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:21:52.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:21:52.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:52.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:53.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:54.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:54.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:54.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:21:54.057 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:21:54.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:54.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:54.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:55.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:55.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:55.567 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:21:55.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:21:55.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:55.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:55.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:56.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:56.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:56.997 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:21:56.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:21:56.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:56.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:57.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:21:58.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:21:58.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:21:58.375 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:21:58.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:21:58.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:21:58.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:21:59.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:00.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:00.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:00.141 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:22:00.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:22:00.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:00.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:00.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:01.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:01.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:01.894 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:22:01.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:22:01.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:01.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:02.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:03.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:03.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:03.393 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:22:03.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:22:03.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:03.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:04.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:05.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:05.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:05.156 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:22:05.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:22:05.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:05.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:05.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:06.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:06.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:06.917 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:22:06.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:22:06.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:06.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:07.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:08.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:08.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:08.712 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:22:08.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:22:08.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:08.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:22:09.098 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/178661,v1:127.0.0.1:6819/178661] [v2:127.0.0.1:6820/178661,v1:127.0.0.1:6821/178661] exists,up 7add4a90-0765-425d-82a3-2d00ccf5a6ea 2024-09-15T04:22:09.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:22:09.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:22:09.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:22:09.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-09-15T04:22:09.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=178661 2024-09-15T04:22:09.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-09-15T04:22:09.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T04:22:09.503 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T04:22:09.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T04:22:09.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T04:22:10.193 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T04:22:10.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:22:11.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T04:22:13.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-09-15T04:22:13.062 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T04:22:13.062 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-09-15T04:22:13.062 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_179640 2024-09-15T04:22:13.063 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T04:22:13.063 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T04:22:14.063 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 80 64 0.249973 0.25 0.260366 0.236371 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 1.90099 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 128 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 4096 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 4096 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.263021 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.25 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.25 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 67 2024-09-15T04:22:15.063 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 0 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 64 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 64 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.236916 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0287034 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.277162 2024-09-15T04:22:15.064 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.150441 2024-09-15T04:22:15.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-09-15T04:22:16.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-09-15T04:22:16.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-09-15T04:22:16.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-15T04:22:16.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-09-15T04:22:16.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=178661 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 178661 -eq 0 ']' 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=178661 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 178661 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-15T04:22:16.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-15T04:22:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-15T04:22:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-15T04:22:17.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-15T04:22:17.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-15T04:22:17.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 178661 -eq 178661 ']' 2024-09-15T04:22:17.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=175151 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 175151 -eq 178661 ']' 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=175151 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 175151 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-09-15T04:22:17.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-09-15T04:22:18.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-09-15T04:22:18.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-09-15T04:22:18.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-09-15T04:22:18.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-09-15T04:22:18.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-09-15T04:22:18.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-09-15T04:22:18.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-15T04:22:18.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-09-15T04:22:18.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-09-15T04:22:18.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-09-15T04:22:18.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-09-15T04:22:18.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-09-15T04:22:18.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-15T04:22:18.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:22:18.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-15T04:22:18.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:22:18.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:22:18.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:22:18.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:22:18.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:22:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:22:19.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:22:19.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:22:19.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:22:19.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:22:19.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:22:19.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:22:19.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:22:19.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:22:19.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:22:19.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:22:19.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-15T04:22:19.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:22:19.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:22:19.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173995 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-fast-mark-down 0 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:22:19.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:22:19.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:22:19.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:22:19.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:22:19.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:22:19.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:22:19.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:22:19.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:22:19.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:22:19.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:22:19.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:22:19.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-09-15T04:22:19.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:22:19.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.173995 2024-09-15T04:22:19.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.173995 2024-09-15T04:22:19.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:22:19.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:22:19.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:22:19.122 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:22:19.123 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:22:19.186 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-09-15T04:22:19.186 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-09-15T04:22:19.249 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:22:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-force-create-pg 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none ' 2024-09-15T04:22:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-09-15T04:22:19.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-09-15T04:22:19.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:22:19.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:22:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:22:19.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:22:19.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:22:19.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:22:19.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:22:19.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:22:19.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:22:19.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:22:19.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:22:19.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:22:19.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:22:19.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:22:19.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:22:19.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:22:19.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:22:19.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-15T04:22:19.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:22:19.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:19.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179803 2024-09-15T04:22:19.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:22:19.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:22:19.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-09-15T04:22:19.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:22:19.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:19.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.179803 2024-09-15T04:22:19.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:22:19.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-09-15T04:22:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T04:22:19.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:22:19.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:22:19.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:22:19.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:22:19.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:19.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:19.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T04:22:19.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:22:19.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.179803/ceph-mon.a.asok 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:22:19.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.179803/ceph-mon.a.asok config get fsid 2024-09-15T04:22:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:22:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:22:19.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:22:19.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:22:19.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:22:19.944 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:22:19.944 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:22:19.944 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.179803/ceph-mon.a.asok 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:22:19.945 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.179803/ceph-mon.a.asok config get mon_host 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-09-15T04:22:20.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:20.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:20.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:22:20.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:22:20.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-09-15T04:22:20.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-15T04:22:20.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:22:20.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:22:20.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:22:20.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:22:20.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 2024-09-15T04:22:20.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4' 2024-09-15T04:22:20.879 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 2024-09-15T04:22:20.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:22:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8YOZmqqzoNBAAppd/pB1+eVXyYd5hcPuRiw== 2024-09-15T04:22:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8YOZmqqzoNBAAppd/pB1+eVXyYd5hcPuRiw=="}' 2024-09-15T04:22:20.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 -i td/osd-force-create-pg/0/new.json 2024-09-15T04:22:21.190 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:22:21.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-09-15T04:22:21.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8YOZmqqzoNBAAppd/pB1+eVXyYd5hcPuRiw== --osd-uuid 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 2024-09-15T04:22:21.224 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.222+0000 7f31bc74f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:21.237 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.235+0000 7f31bc74f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:21.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.237+0000 7f31bc74f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:21.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.237+0000 7f31bc74f440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-15T04:22:21.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.237+0000 7f31bc74f440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-15T04:22:21.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.237+0000 7f31bc74f440 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-09-15T04:22:21.277 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:21.276+0000 7f31bc74f440 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-09-15T04:22:26.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-09-15T04:22:26.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:22:26.628 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:22:26.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:22:26.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:22:27.114 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:22:27.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:22:27.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:22:27.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:22:27.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:22:27.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:22:27.135 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:27.134+0000 7fb12ae30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:27.136 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:27.136+0000 7fb12ae30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:27.137 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:27.137+0000 7fb12ae30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:27.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:22:27.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:22:27.466 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:22:27.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:27.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:27.652 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:27.652+0000 7fb12ae30440 -1 Falling back to public interface 2024-09-15T04:22:27.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:28.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:28.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:28.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:22:28.831 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:22:28.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:28.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:29.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:30.118 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:30.117+0000 7fb12ae30440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:22:30.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:30.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:30.187 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:22:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:22:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:30.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:30.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:31.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:31.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:31.586 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:22:31.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:22:31.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:31.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:33.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:33.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:33.100 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:22:33.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:22:33.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:33.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:33.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:34.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:34.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:34.483 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:22:34.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:22:34.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:34.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:34.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:35.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:35.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:35.911 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:22:35.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:22:35.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:35.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:36.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:37.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:37.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:37.307 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:22:37.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:22:37.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:37.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:38.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:38.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:38.700 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:22:38.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:22:38.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:38.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:39.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:40.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:40.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:40.169 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:22:40.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:22:40.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:40.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:40.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:41.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:41.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:41.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:22:41.549 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:22:41.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:41.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:42.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:43.158 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:22:43.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:22:43.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:43.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:43.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:44.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:44.620 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:22:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:22:44.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:44.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:45.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:46.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:46.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:46.409 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:22:46.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:22:46.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:46.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:46.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:47.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:47.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:47.950 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:22:47.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:22:47.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:47.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:48.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:49.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:49.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:49.580 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:22:49.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:22:49.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:49.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:50.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:51.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:51.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:51.212 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:22:51.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:22:51.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:51.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:51.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:22:52.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:22:52.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:22:52.793 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:22:52.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:22:52.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:22:52.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:22:53.147 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/180894,v1:127.0.0.1:6803/180894] [v2:127.0.0.1:6804/180894,v1:127.0.0.1:6805/180894] exists,up 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:22:53.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:53.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-09-15T04:22:53.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:22:53.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97ac4978-3b32-4c09-a7df-87dc4f16953a 2024-09-15T04:22:53.154 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 97ac4978-3b32-4c09-a7df-87dc4f16953a 2024-09-15T04:22:53.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97ac4978-3b32-4c09-a7df-87dc4f16953a' 2024-09-15T04:22:53.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:22:53.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdYeZmN/O2CRAA4CdrVOfWvq3iWtHNejj4UA== 2024-09-15T04:22:53.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdYeZmN/O2CRAA4CdrVOfWvq3iWtHNejj4UA=="}' 2024-09-15T04:22:53.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97ac4978-3b32-4c09-a7df-87dc4f16953a -i td/osd-force-create-pg/1/new.json 2024-09-15T04:22:53.794 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:22:53.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-09-15T04:22:53.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdYeZmN/O2CRAA4CdrVOfWvq3iWtHNejj4UA== --osd-uuid 97ac4978-3b32-4c09-a7df-87dc4f16953a 2024-09-15T04:22:53.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.827+0000 7f044963b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:53.852 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.851+0000 7f044963b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:53.853 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.852+0000 7f044963b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:22:53.854 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.853+0000 7f044963b440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-15T04:22:53.854 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.853+0000 7f044963b440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-15T04:22:53.854 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.853+0000 7f044963b440 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-09-15T04:22:53.935 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:22:53.934+0000 7f044963b440 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-09-15T04:22:59.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-09-15T04:22:59.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:22:59.895 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:22:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:22:59.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:23:00.376 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:23:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:23:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:23:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:23:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:23:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:23:00.397 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:00.396+0000 7ff7a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:00.399 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:00.398+0000 7ff7a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:00.400 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:00.399+0000 7ff7a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:00.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:23:00.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:23:00.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:00.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:00.915 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:00.915+0000 7ff7a0521440 -1 Falling back to public interface 2024-09-15T04:23:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:02.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:02.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:02.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:23:02.101 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:23:02.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:02.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:02.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:03.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:03.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:03.483 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:23:03.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:23:03.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:03.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:03.497 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:03.496+0000 7ff7a0521440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:23:03.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:04.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:04.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:04.957 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:23:04.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:23:04.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:04.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:05.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:06.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:06.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:06.388 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:23:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:23:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:06.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:07.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:07.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:07.746 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:23:07.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:23:07.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:07.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:08.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:09.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:09.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:09.101 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:23:09.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:23:09.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:09.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:09.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:10.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:10.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:10.528 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:23:10.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:23:10.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:10.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:10.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:11.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:11.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:11.885 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:23:11.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:23:11.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:11.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:12.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:13.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:13.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:13.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:23:13.381 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:23:13.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:13.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:13.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:14.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:14.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:14.745 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:23:14.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:23:14.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:14.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:15.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:16.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:16.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:16.237 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:23:16.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:23:16.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:16.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:16.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:17.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:17.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:17.962 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:23:17.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:23:17.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:17.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:18.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:19.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:19.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:19.767 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:23:19.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:23:19.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:19.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:20.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:21.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:21.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:21.389 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:23:21.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:23:21.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:21.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:22.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:23.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:23.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:23.357 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:23:23.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:23:23.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:23.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:24.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:25.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:25.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:25.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:23:25.126 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:23:25.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:25.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:26.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:27.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:27.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:27.087 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:23:27.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:23:27.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:27.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:23:27.552 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/182648,v1:127.0.0.1:6811/182648] [v2:127.0.0.1:6812/182648,v1:127.0.0.1:6813/182648] exists,up 97ac4978-3b32-4c09-a7df-87dc4f16953a 2024-09-15T04:23:27.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:23:27.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-15T04:23:27.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:23:27.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=40dd6f39-c418-4296-8166-39feb83bcc86 2024-09-15T04:23:27.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 40dd6f39-c418-4296-8166-39feb83bcc86' 2024-09-15T04:23:27.560 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 40dd6f39-c418-4296-8166-39feb83bcc86 2024-09-15T04:23:27.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:23:27.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/YeZmBoK0IRAAVbCJFZjzjqW7mnRWjiAzIw== 2024-09-15T04:23:27.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/YeZmBoK0IRAAVbCJFZjzjqW7mnRWjiAzIw=="}' 2024-09-15T04:23:27.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 40dd6f39-c418-4296-8166-39feb83bcc86 -i td/osd-force-create-pg/2/new.json 2024-09-15T04:23:28.008 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:23:28.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-09-15T04:23:28.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/YeZmBoK0IRAAVbCJFZjzjqW7mnRWjiAzIw== --osd-uuid 40dd6f39-c418-4296-8166-39feb83bcc86 2024-09-15T04:23:28.042 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.041+0000 7f3d46237440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:28.059 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.059+0000 7f3d46237440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:28.060 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.060+0000 7f3d46237440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:28.061 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.060+0000 7f3d46237440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-15T04:23:28.061 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.060+0000 7f3d46237440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-15T04:23:28.061 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.060+0000 7f3d46237440 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-09-15T04:23:28.099 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:28.099+0000 7f3d46237440 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-09-15T04:23:34.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-09-15T04:23:34.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:23:34.308 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:23:34.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:23:34.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:23:34.790 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:23:34.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:23:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:23:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:23:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:23:34.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:23:34.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:34.810+0000 7fa667a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:34.813 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:34.812+0000 7fa667a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:34.814 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:34.813+0000 7fa667a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:23:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:23:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:23:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:23:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:23:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:23:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:35.140 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:23:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:23:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:35.328 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:35.328+0000 7fa667a0b440 -1 Falling back to public interface 2024-09-15T04:23:35.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:36.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:36.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:36.547 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:23:36.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:23:36.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:36.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:36.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:37.828 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:23:37.827+0000 7fa667a0b440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:23:37.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:37.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:37.902 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:23:37.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:23:37.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:37.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:38.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:39.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:39.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:39.617 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:23:39.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:23:39.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:39.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:40.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:41.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:41.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:41.009 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:23:41.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:23:41.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:41.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:41.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:42.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:42.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:42.418 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:23:42.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:23:42.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:42.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:42.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:43.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:43.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:43.849 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:23:43.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:23:43.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:43.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:44.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:45.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:45.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:45.221 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:23:45.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:23:45.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:45.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:45.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:46.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:46.653 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:23:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:23:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:47.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:48.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:48.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:48.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:23:48.033 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:23:48.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:48.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:48.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:49.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:49.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:49.391 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:23:49.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:23:49.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:49.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:49.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:50.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:50.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:50.762 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:23:50.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:23:50.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:50.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:51.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:52.137 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:23:52.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:23:52.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:52.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:53.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:53.566 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:23:53.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:53.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:23:53.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:53.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:54.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:55.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:55.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:55.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:23:55.249 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:23:55.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:55.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:56.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:57.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:57.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:57.452 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:23:57.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:23:57.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:57.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:23:58.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:23:59.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:23:59.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:23:59.133 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:23:59.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:23:59.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:23:59.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:00.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:01.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:01.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:01.098 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:24:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:24:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:01.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:02.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:02.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:02.902 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:24:02.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:24:02.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:02.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:03.279 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/184403,v1:127.0.0.1:6819/184403] [v2:127.0.0.1:6820/184403,v1:127.0.0.1:6821/184403] exists,up 40dd6f39-c418-4296-8166-39feb83bcc86 2024-09-15T04:24:03.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:24:03.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:24:03.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:24:03.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-09-15T04:24:03.949 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T04:24:03.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-09-15T04:24:03.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:24:03.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:24:03.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:24:03.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:24:04.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:24:04.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:24:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:24:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T04:24:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:24:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:04.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:24:04.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836498 2024-09-15T04:24:04.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836498 2024-09-15T04:24:04.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498' 2024-09-15T04:24:04.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:04.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:24:05.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672973 2024-09-15T04:24:05.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672973 2024-09-15T04:24:05.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672973' 2024-09-15T04:24:05.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:05.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:24:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509447 2024-09-15T04:24:05.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509447 2024-09-15T04:24:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836498 1-42949672973 2-64424509447' 2024-09-15T04:24:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:05.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836498 2024-09-15T04:24:05.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:05.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:24:05.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836498 2024-09-15T04:24:05.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:05.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836498 2024-09-15T04:24:05.241 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836498 2024-09-15T04:24:05.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2024-09-15T04:24:05.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:24:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836496 -lt 21474836498 2024-09-15T04:24:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:24:06.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:24:06.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:24:07.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836498 2024-09-15T04:24:07.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:07.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672973 2024-09-15T04:24:07.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:07.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:24:07.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672973 2024-09-15T04:24:07.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:07.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672973 2024-09-15T04:24:07.049 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672973 2024-09-15T04:24:07.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-09-15T04:24:07.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:24:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672973 -lt 42949672973 2024-09-15T04:24:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:07.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509447 2024-09-15T04:24:07.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:07.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:24:07.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509447 2024-09-15T04:24:07.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509447 2024-09-15T04:24:07.395 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509447 2024-09-15T04:24:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-09-15T04:24:07.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:24:07.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509447 -lt 64424509447 2024-09-15T04:24:07.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:24:07.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:07.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:24:08.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:24:08.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:24:08.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=37 2024-09-15T04:24:08.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:24:08.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:08.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 37 = 50 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 37 '!=' -1 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=37 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:24:09.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:24:09.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:24:09.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:24:09.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:24:09.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:24:09.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:24:09.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:24:09.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:24:09.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:24:09.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=37 2024-09-15T04:24:09.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:24:09.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:09.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 37 = 50 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 37 '!=' 37 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:24:09.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:24:09.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:24:09.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:24:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T04:24:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T04:24:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T04:24:10.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:24:10.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T04:24:10.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:24:10.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:24:10.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:24:10.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:24:10.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:24:10.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:24:10.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:24:10.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:24:10.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-15T04:24:10.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:24:10.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:10.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:11.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-15T04:24:11.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:11.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-09-15T04:24:11.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:11.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:11.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:11.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:12.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:12.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:12.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:12.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-09-15T04:24:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:PG '1.0' not found 2024-09-15T04:24:13.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-09-15T04:24:15.316 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T04:24:15.316 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T04:24:15.316 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T04:24:15.412 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T04:24:15.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-09-15T04:24:17.237 INFO:tasks.workunit.client.0.smithi138.stderr:PG '1.0' not found 2024-09-15T04:24:17.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-09-15T04:24:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-15T04:24:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:24:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:24:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:24:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-09-15T04:24:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:24:17.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-09-15T04:24:17.508 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:24:17.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:24:17.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:24:17.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-09-15T04:24:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:24:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:24:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:24:17.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:24:17.528 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:17.527+0000 7f9450017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:17.530+0000 7f9450017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:17.531+0000 7f9450017440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:17.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:24:17.861 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:24:17.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:17.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:18.049 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:18.049+0000 7f9450017440 -1 Falling back to public interface 2024-09-15T04:24:18.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:19.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:19.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:19.210 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:24:19.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:24:19.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:19.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:19.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:20.528 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:20.527+0000 7f9450017440 -1 osd.0 22 log_to_monitors true 2024-09-15T04:24:20.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:20.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:20.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:24:20.564 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:24:20.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:20.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:20.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:21.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:21.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:21.974 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:24:21.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:24:21.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:21.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:22.314 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 27 up_thru 27 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6802/187022,v1:127.0.0.1:6803/187022] [v2:127.0.0.1:6804/187022,v1:127.0.0.1:6805/187022] exists,up 6f3e54bd-c9df-4e3f-9500-e95e3aa39ed4 2024-09-15T04:24:22.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:24:22.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:24:22.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:24:22.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-09-15T04:24:22.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:24:22.318 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:24:22.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:24:22.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:24:22.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:24:22.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:24:22.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-09-15T04:24:22.320 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:24:22.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:24:22.337 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:22.336+0000 7fdd21f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:22.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:22.351+0000 7fdd21f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:22.352 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:22.352+0000 7fdd21f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:22.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T04:24:22.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:24:22.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:24:22.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:22.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:24:22.868 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:22.868+0000 7fdd21f2d440 -1 Falling back to public interface 2024-09-15T04:24:23.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:24.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:24.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:24.033 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:24:24.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:24:24.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:24.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:24:24.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:25.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:25.425 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:24:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:24:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:25.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:24:25.512 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:25.512+0000 7fdd21f2d440 -1 osd.1 23 log_to_monitors true 2024-09-15T04:24:25.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:26.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:26.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:26.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:24:26.803 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:24:26.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:26.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:24:27.260 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 30 up_thru 30 down_at 24 last_clean_interval [10,23) [v2:127.0.0.1:6810/187600,v1:127.0.0.1:6811/187600] [v2:127.0.0.1:6812/187600,v1:127.0.0.1:6813/187600] exists,up 97ac4978-3b32-4c09-a7df-87dc4f16953a 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:24:27.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-09-15T04:24:27.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:24:27.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:24:27.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:24:27.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:24:27.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:24:27.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-09-15T04:24:27.267 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=838b16e3-ca7d-4eda-8c8f-1b874c1e7df5 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.179803/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:24:27.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:24:27.284 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:27.283+0000 7f2dba11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:27.321 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:27.319+0000 7f2dba11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:27.321 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:27.320+0000 7f2dba11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:27.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T04:24:27.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:24:27.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:24:27.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:24:27.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:24:27.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:27.640 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:24:27.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:24:27.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:27.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:27.844 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:27.844+0000 7f2dba11d440 -1 Falling back to public interface 2024-09-15T04:24:27.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:28.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:28.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:28.989 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:24:28.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:24:28.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:28.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:29.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:30.301+0000 7f2dba11d440 -1 osd.2 24 log_to_monitors true 2024-09-15T04:24:30.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:24:30.344 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:24:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:30.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:31.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:31.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:31.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:24:31.798 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:24:31.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:31.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:24:32.152 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 33 up_thru 33 down_at 25 last_clean_interval [15,24) [v2:127.0.0.1:6818/188178,v1:127.0.0.1:6819/188178] [v2:127.0.0.1:6820/188178,v1:127.0.0.1:6821/188178] exists,up 40dd6f39-c418-4296-8166-39feb83bcc86 2024-09-15T04:24:32.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:24:32.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:24:32.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:24:32.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-09-15T04:24:42.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-09-15T04:24:42.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-09-15T04:24:42.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-09-15T04:24:42.504 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 37s 0'0 20:10 [1]p1 [1]p1 2024-09-15T04:24:03.570289+0000 2024-09-15T04:24:03.570289+0000 0 periodic scrub scheduled @ 2024-09-16T13:56:27.644552+0000 2024-09-15T04:24:42.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-09-15T04:24:43.113 INFO:tasks.workunit.client.0.smithi138.stderr:pg 1.0 already creating 2024-09-15T04:24:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-09-15T04:24:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:24:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:24:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:24:43.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:24:43.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:24:43.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:24:43.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:24:43.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:24:43.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:24:43.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:24:43.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:24:43.569 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:24:43.569 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T04:24:43.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:24:43.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:43.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:24:43.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=115964116999 2024-09-15T04:24:43.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 115964116999 2024-09-15T04:24:43.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999' 2024-09-15T04:24:43.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:43.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:24:43.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018886 2024-09-15T04:24:43.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018886 2024-09-15T04:24:43.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999 1-128849018886' 2024-09-15T04:24:43.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:24:43.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:24:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920773 2024-09-15T04:24:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920773 2024-09-15T04:24:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-115964116999 1-128849018886 2-141733920773' 2024-09-15T04:24:43.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:43.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-115964116999 2024-09-15T04:24:43.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:43.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:24:43.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-115964116999 2024-09-15T04:24:43.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:43.899 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 115964116999 2024-09-15T04:24:43.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=115964116999 2024-09-15T04:24:43.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 115964116999' 2024-09-15T04:24:43.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:24:44.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 115964116997 -lt 115964116999 2024-09-15T04:24:44.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:24:45.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:24:45.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:24:45.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 115964116999 -lt 115964116999 2024-09-15T04:24:45.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:45.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-128849018886 2024-09-15T04:24:45.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:45.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:24:45.593 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-128849018886 2024-09-15T04:24:45.593 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:45.594 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 128849018886 2024-09-15T04:24:45.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018886 2024-09-15T04:24:45.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 128849018886' 2024-09-15T04:24:45.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:24:45.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018886 -lt 128849018886 2024-09-15T04:24:45.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:24:45.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-141733920773 2024-09-15T04:24:45.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:24:45.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:24:45.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-141733920773 2024-09-15T04:24:45.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:24:45.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920773 2024-09-15T04:24:45.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 141733920773' 2024-09-15T04:24:45.964 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 141733920773 2024-09-15T04:24:45.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:24:46.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920773 -lt 141733920773 2024-09-15T04:24:46.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:24:46.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:46.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:46.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-09-15T04:24:46.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:24:46.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:24:47.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-09-15T04:24:47.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:24:47.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:24:47.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:47.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:47.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:47.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:47.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:24:47.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:24:47.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:24:47.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:24:47.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:24:47.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:24:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:24:47.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:24:47.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:24:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:24:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-15T04:24:47.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:24:47.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:47.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:24:47.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179803 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-force-create-pg 0 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:47.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:47.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:47.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:47.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:24:47.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:24:47.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:24:47.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:24:47.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:24:47.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:24:47.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:24:47.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:24:47.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:24:47.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:24:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-09-15T04:24:47.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:24:47.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:47.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.179803 2024-09-15T04:24:47.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.179803 2024-09-15T04:24:47.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:24:47.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:24:47.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:24:47.741 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:24:47.741 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:24:47.805 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-09-15T04:24:47.806 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-09-15T04:24:47.867 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:24:47.872 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:24:47.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:24:47.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-markdown 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-09-15T04:24:47.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none ' 2024-09-15T04:24:47.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-09-15T04:24:47.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-09-15T04:24:47.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:24:47.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-09-15T04:24:47.875 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_markdown_boot_exceed_time 2024-09-15T04:24:47.875 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_markdown_exceed_maxdown_count 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_osd_stop' 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:24:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:24:47.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:24:47.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:24:47.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:24:47.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:24:47.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:24:47.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:24:47.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:24:47.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:24:47.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:24:47.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:24:47.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:24:47.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:24:47.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:24:47.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:24:47.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:24:47.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:24:47.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:47.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:47.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:24:47.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:24:47.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:24:47.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-15T04:24:47.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:24:47.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:47.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:47.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T04:24:47.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-15T04:24:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-15T04:24:48.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:24:48.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:48.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:48.240 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:48.241 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:48.241 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:48.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:24:48.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:24:48.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:24:48.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:24:48.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:24:48.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:48.494 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:24:48.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:24:48.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T04:24:48.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:24:48.569 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:24:48.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:24:48.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-15T04:24:48.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:24:49.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:24:49.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:24:49.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:24:49.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:24:49.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-15T04:24:49.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:24:49.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3a98a46a-6a1a-4ef8-8bf7-03e5c894a473 2024-09-15T04:24:49.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3a98a46a-6a1a-4ef8-8bf7-03e5c894a473' 2024-09-15T04:24:49.503 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 3a98a46a-6a1a-4ef8-8bf7-03e5c894a473 2024-09-15T04:24:49.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:24:49.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRYeZmj0uXHhAA5pH7xkGl0vXxxoRTFVi44Q== 2024-09-15T04:24:49.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRYeZmj0uXHhAA5pH7xkGl0vXxxoRTFVi44Q=="}' 2024-09-15T04:24:49.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3a98a46a-6a1a-4ef8-8bf7-03e5c894a473 -i td/osd-markdown/0/new.json 2024-09-15T04:24:49.781 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:24:49.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-15T04:24:49.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCRYeZmj0uXHhAA5pH7xkGl0vXxxoRTFVi44Q== --osd-uuid 3a98a46a-6a1a-4ef8-8bf7-03e5c894a473 2024-09-15T04:24:49.814 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.813+0000 7fef08015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:49.827 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.826+0000 7fef08015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:49.828 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.827+0000 7fef08015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:49.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.828+0000 7fef08015440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:24:49.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.828+0000 7fef08015440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:24:49.829 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.828+0000 7fef08015440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:24:49.914 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:49.913+0000 7fef08015440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-15T04:24:55.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-15T04:24:55.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:24:55.238 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:24:55.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:24:55.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:24:55.724 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:24:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:24:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:24:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:24:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:24:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:24:55.745 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:55.744+0000 7fbba240a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:55.747 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:55.747+0000 7fbba240a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:55.748 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:55.748+0000 7fbba240a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:24:56.070 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:24:56.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:56.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:56.278 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:56.278+0000 7fbba240a440 -1 Falling back to public interface 2024-09-15T04:24:56.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:57.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:57.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:57.424 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:24:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:24:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:57.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:24:58.762 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:24:58.761+0000 7fbba240a440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:24:58.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:24:58.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:24:58.780 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:24:58.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:24:58.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:24:58.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:24:59.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:00.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:00.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:00.153 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:25:00.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:25:00.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:00.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:00.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:01.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:01.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:01.627 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:25:01.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:25:01.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:01.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:02.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:03.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:03.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:03.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:25:03.026 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:25:03.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:03.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:04.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:04.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:04.418 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:25:04.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:25:04.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:04.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:04.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:05.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:05.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:05.795 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:25:05.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:25:05.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:05.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:06.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:07.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:07.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:07.152 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:25:07.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:25:07.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:07.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:07.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:08.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:08.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:08.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:25:08.572 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:25:08.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:08.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:08.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:09.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:09.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:09.929 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:25:09.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:25:09.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:09.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:10.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:11.286 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:25:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:25:11.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:11.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:12.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:13.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:13.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:13.146 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:25:13.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:25:13.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:13.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:13.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:14.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:14.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:14.765 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:25:14.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:25:14.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:14.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:15.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:16.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:16.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:16.358 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:25:16.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:25:16.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:16.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:18.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:18.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:18.178 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:25:18.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:25:18.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:18.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:19.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:20.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:20.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:20.043 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:25:20.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:25:20.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:20.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:21.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:21.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:21.871 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:25:21.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:25:21.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:21.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:25:22.226 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/190298,v1:127.0.0.1:6803/190298] [v2:127.0.0.1:6804/190298,v1:127.0.0.1:6805/190298] exists,up 3a98a46a-6a1a-4ef8-8bf7-03e5c894a473 2024-09-15T04:25:22.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:25:22.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:25:22.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:25:22.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-09-15T04:25:22.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:25:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:25:22.232 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:25:22.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:25:22.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:25:22.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:25:22.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-15T04:25:22.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:25:22.234 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 56dc8826-75b0-45d0-bac0-74de87ca5794 2024-09-15T04:25:22.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=56dc8826-75b0-45d0-bac0-74de87ca5794 2024-09-15T04:25:22.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 56dc8826-75b0-45d0-bac0-74de87ca5794' 2024-09-15T04:25:22.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:25:22.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyYeZmMPFkDhAA31HWxHDm/r6S123J34q7KA== 2024-09-15T04:25:22.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyYeZmMPFkDhAA31HWxHDm/r6S123J34q7KA=="}' 2024-09-15T04:25:22.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 56dc8826-75b0-45d0-bac0-74de87ca5794 -i td/osd-markdown/1/new.json 2024-09-15T04:25:22.736 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:25:22.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-15T04:25:22.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyYeZmMPFkDhAA31HWxHDm/r6S123J34q7KA== --osd-uuid 56dc8826-75b0-45d0-bac0-74de87ca5794 2024-09-15T04:25:22.770 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.769+0000 7fa53580e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:22.771 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.771+0000 7fa53580e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:22.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.772+0000 7fa53580e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:22.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.772+0000 7fa53580e440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:25:22.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.772+0000 7fa53580e440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:25:22.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.772+0000 7fa53580e440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:25:22.801 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:22.800+0000 7fa53580e440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-15T04:25:28.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-15T04:25:28.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:25:28.982 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:25:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:25:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:25:29.466 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:25:29.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:25:29.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:25:29.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:25:29.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:25:29.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:25:29.486 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:29.485+0000 7f59be522440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:29.488 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:29.487+0000 7f59be522440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:29.489 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:29.488+0000 7f59be522440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:29.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:25:29.818 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:25:29.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:29.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:30.010 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:30.009+0000 7f59be522440 -1 Falling back to public interface 2024-09-15T04:25:30.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:31.182 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:25:31.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:25:31.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:31.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:31.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:32.493+0000 7f59be522440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:25:32.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:32.535 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:25:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:25:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:32.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:33.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:34.173 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:25:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:25:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:34.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:35.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:35.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:35.594 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:25:35.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:25:35.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:35.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:35.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:36.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:36.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:36.992 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:25:36.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:25:36.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:36.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:37.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:38.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:38.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:38.456 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:25:38.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:25:38.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:38.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:38.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:39.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:39.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:39.812 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:25:39.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:25:39.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:39.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:40.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:41.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:41.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:41.296 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:25:41.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:25:41.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:41.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:41.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:42.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:42.651 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:25:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:25:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:43.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:44.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:44.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:44.029 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:25:44.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:25:44.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:44.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:44.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:45.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:45.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:45.484 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:25:45.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:25:45.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:45.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:46.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:47.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:25:47.515 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:25:47.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:47.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:48.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:49.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:49.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:49.335 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:25:49.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:25:49.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:49.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:50.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:50.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:50.778 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:25:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:25:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:51.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:52.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:52.297 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:25:52.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:25:52.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:52.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:53.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:54.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:54.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:54.049 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:25:54.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:25:54.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:54.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:54.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:55.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:55.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:55.792 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:25:55.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:25:55.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:55.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:56.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:25:57.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:25:57.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:25:57.262 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:25:57.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:25:57.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:25:57.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:25:57.641 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/192052,v1:127.0.0.1:6811/192052] [v2:127.0.0.1:6812/192052,v1:127.0.0.1:6813/192052] exists,up 56dc8826-75b0-45d0-bac0-74de87ca5794 2024-09-15T04:25:57.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:25:57.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:25:57.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-15T04:25:57.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:25:57.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2 2024-09-15T04:25:57.649 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2 2024-09-15T04:25:57.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2' 2024-09-15T04:25:57.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:25:57.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVYeZmBXIzJxAASdaeFgYi/Lt1cs35EZe94A== 2024-09-15T04:25:57.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVYeZmBXIzJxAASdaeFgYi/Lt1cs35EZe94A=="}' 2024-09-15T04:25:57.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2 -i td/osd-markdown/2/new.json 2024-09-15T04:25:58.151 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:25:58.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-15T04:25:58.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVYeZmBXIzJxAASdaeFgYi/Lt1cs35EZe94A== --osd-uuid e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2 2024-09-15T04:25:58.185 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.184+0000 7f8457628440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:58.187 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.186+0000 7f8457628440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:58.188 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.187+0000 7f8457628440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:25:58.188 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.188+0000 7f8457628440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:25:58.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.188+0000 7f8457628440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:25:58.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.188+0000 7f8457628440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:25:58.233 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:25:58.232+0000 7f8457628440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-15T04:26:04.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-15T04:26:04.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:26:04.187 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:26:04.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:26:04.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:26:04.668 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:26:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:26:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:26:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:26:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:26:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:26:04.690 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:26:04.689+0000 7f173d611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:26:04.692 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:26:04.691+0000 7f173d611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:26:04.693 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:26:04.692+0000 7f173d611440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:05.014 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:26:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:26:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:05.212 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:26:05.212+0000 7f173d611440 -1 Falling back to public interface 2024-09-15T04:26:05.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:06.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:06.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:06.373 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:26:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:26:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:06.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:07.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:07.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:07.725 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:26:07.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:26:07.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:07.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:07.828 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:26:07.827+0000 7f173d611440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:26:08.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:09.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:09.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:09.146 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:26:09.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:26:09.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:09.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:09.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:10.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:10.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:10.632 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:26:10.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:26:10.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:10.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:11.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:12.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:12.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:12.024 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:26:12.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:26:12.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:12.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:12.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:13.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:13.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:13.422 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:26:13.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:26:13.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:13.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:13.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:14.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:14.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:14.897 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:26:14.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:26:14.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:14.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:15.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:16.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:16.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:16.318 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:26:16.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:26:16.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:16.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:16.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:17.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:17.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:17.740 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:26:17.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:26:17.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:17.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:18.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:19.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:19.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:19.252 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:26:19.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:26:19.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:19.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:19.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:20.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:20.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:20.768 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:26:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:26:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:21.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:22.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:22.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:22.352 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:26:22.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:26:22.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:22.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:23.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:24.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:26:24.116 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:26:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:24.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:25.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:25.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:25.832 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:26:25.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:26:25.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:25.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:26.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:27.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:27.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:27.831 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:26:27.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:26:27.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:27.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:28.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:29.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:29.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:29.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:26:29.483 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:26:29.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:29.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:29.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:26:30.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:26:30.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:26:30.982 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:26:30.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:26:30.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:26:30.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:26:31.337 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/193831,v1:127.0.0.1:6819/193831] [v2:127.0.0.1:6820/193831,v1:127.0.0.1:6821/193831] exists,up e142bcc9-9022-4dcf-b2dc-fbbaad7ce2c2 2024-09-15T04:26:31.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:26:31.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:26:31.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:26:31.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-09-15T04:26:31.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T04:26:31.710 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T04:26:31.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T04:26:31.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T04:26:32.345 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T04:26:32.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:26:33.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T04:26:35.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-09-15T04:26:35.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-09-15T04:26:35.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-09-15T04:26:35.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-15T04:26:35.454 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:26:35.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-09-15T04:26:35.608 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:26:35.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-09-15T04:26:35.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-09-15T04:26:35.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-09-15T04:26:35.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-15T04:26:35.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-09-15T04:26:35.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:26:35.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:26:35.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:26:36.085 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:26:36.085 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:26:36.085 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:26:36.085 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:36.085 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:26:36.086 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:26:36.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:26:36.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:26:36.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:26:36.449 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:36.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:26:36.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:26:36.865 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:26:36.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:26:46.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:26:46.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:26:46.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:26:47.357 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:26:47.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:26:47.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:26:47.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:26:47.722 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:47.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:26:47.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:26:48.113 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:26:48.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:26:58.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:26:58.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:26:58.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:26:58.575 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:26:58.575 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:26:58.576 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:26:58.576 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:58.576 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:26:58.576 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:26:58.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:26:58.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:26:58.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:26:58.944 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:26:58.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:26:58.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:26:59.410 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:26:59.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:27:09.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-09-15T04:27:24.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:27:24.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-09-15T04:27:24.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-09-15T04:27:24.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-09-15T04:27:24.884 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:27:24.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-15T04:27:24.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:27:24.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:27:24.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:27:24.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:27:24.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:27:24.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:27:24.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:27:24.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:27:24.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:27:24.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:27:24.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:27:24.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:27:24.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:27:24.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:27:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:27:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:27:25.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:27:25.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:27:25.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:27:25.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:27:25.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:27:25.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:27:25.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:27:25.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:27:25.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:27:25.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:27:25.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:27:25.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:27:25.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:27:25.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:27:25.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:27:25.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:27:25.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:27:25.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:27:25.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:27:25.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:27:25.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:27:25.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-15T04:27:25.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:27:25.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-09-15T04:27:25.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-15T04:27:25.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:27:25.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:27:25.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:27:25.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-15T04:27:25.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.384 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:25.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:27:25.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:27:25.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:27:25.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:27:25.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T04:27:25.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:27:25.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:27:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T04:27:25.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-09-15T04:27:25.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-15T04:27:25.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:27:25.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:27:25.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:27:25.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-15T04:27:25.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:27:26.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:27:26.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:27:26.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:27:26.639 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:27:26.639 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:26.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:26.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:27:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:27:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:26.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:27:26.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7b49a7b-7a91-4659-87a1-15d456dec619 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f7b49a7b-7a91-4659-87a1-15d456dec619' 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 f7b49a7b-7a91-4659-87a1-15d456dec619 2024-09-15T04:27:26.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:27:26.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAuYuZmt1FdKBAALxvJ9C0CNJGapPFzTgXcRg== 2024-09-15T04:27:26.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAuYuZmt1FdKBAALxvJ9C0CNJGapPFzTgXcRg=="}' 2024-09-15T04:27:26.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7b49a7b-7a91-4659-87a1-15d456dec619 -i td/osd-markdown/0/new.json 2024-09-15T04:27:26.983 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:27:26.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-15T04:27:26.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAuYuZmt1FdKBAALxvJ9C0CNJGapPFzTgXcRg== --osd-uuid f7b49a7b-7a91-4659-87a1-15d456dec619 2024-09-15T04:27:27.016 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.016+0000 7f6a44f60440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:27.029 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.029+0000 7f6a44f60440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:27.030 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.030+0000 7f6a44f60440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.030+0000 7f6a44f60440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:27:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.030+0000 7f6a44f60440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:27:27.031 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.030+0000 7f6a44f60440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:27:27.072 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:27.069+0000 7f6a44f60440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-15T04:27:32.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-15T04:27:32.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:27:32.434 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:27:32.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:27:32.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:27:32.919 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:27:32.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:27:32.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:27:32.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:27:32.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:27:32.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:27:32.939 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:32.939+0000 7f6a3f68d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:32.941 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:32.940+0000 7f6a3f68d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:32.942 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:32.941+0000 7f6a3f68d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:27:33.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:33.267 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:27:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:27:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:33.458 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:33.458+0000 7f6a3f68d440 -1 Falling back to public interface 2024-09-15T04:27:33.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:34.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:34.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:34.629 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:27:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:27:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:35.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:35.934 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:27:35.933+0000 7f6a3f68d440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:27:36.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:36.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:36.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:27:36.030 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:27:36.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:36.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:36.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:37.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:37.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:37.538 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:27:37.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:27:37.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:37.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:37.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:38.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:38.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:38.899 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:27:38.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:27:38.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:38.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:39.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:40.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:40.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:40.281 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:27:40.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:27:40.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:40.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:40.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:41.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:41.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:41.763 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:27:41.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:27:41.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:41.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:42.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:43.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:43.157 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:27:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:27:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:43.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:43.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:44.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:44.619 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:27:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:27:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:44.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:45.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:46.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:46.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:46.007 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:27:46.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:27:46.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:46.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:46.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:47.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:47.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:47.389 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:27:47.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:27:47.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:47.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:48.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:49.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:49.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:49.171 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:27:49.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:27:49.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:49.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:49.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:50.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:50.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:50.679 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:27:50.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:27:50.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:50.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:51.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:52.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:52.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:52.359 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:27:52.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:27:52.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:52.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:53.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:54.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:54.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:54.095 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:27:54.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:27:54.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:54.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:54.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:55.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:55.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:55.636 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:27:55.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:27:55.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:55.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:56.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:57.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:57.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:57.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:27:57.329 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:27:57.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:57.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:58.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:27:59.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:27:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:27:59.090 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:27:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:27:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:27:59.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:27:59.438 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/196308,v1:127.0.0.1:6803/196308] [v2:127.0.0.1:6804/196308,v1:127.0.0.1:6805/196308] exists,up f7b49a7b-7a91-4659-87a1-15d456dec619 2024-09-15T04:27:59.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:27:59.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:27:59.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:27:59.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5c01a9e6-3d18-4509-a1a1-3676688ec050 2024-09-15T04:27:59.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5c01a9e6-3d18-4509-a1a1-3676688ec050' 2024-09-15T04:27:59.445 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 5c01a9e6-3d18-4509-a1a1-3676688ec050 2024-09-15T04:27:59.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:27:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPYuZmzfYYGxAAYi1NHQ9fTMZ5Q5rOs0nfow== 2024-09-15T04:27:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPYuZmzfYYGxAAYi1NHQ9fTMZ5Q5rOs0nfow=="}' 2024-09-15T04:27:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5c01a9e6-3d18-4509-a1a1-3676688ec050 -i td/osd-markdown/1/new.json 2024-09-15T04:28:00.060 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:28:00.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-15T04:28:00.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPYuZmzfYYGxAAYi1NHQ9fTMZ5Q5rOs0nfow== --osd-uuid 5c01a9e6-3d18-4509-a1a1-3676688ec050 2024-09-15T04:28:00.094 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.093+0000 7fb8f0362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:00.095 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.095+0000 7fb8f0362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:00.096 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.096+0000 7fb8f0362440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:00.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.096+0000 7fb8f0362440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:28:00.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.096+0000 7fb8f0362440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:28:00.097 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.096+0000 7fb8f0362440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:28:00.141 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:00.141+0000 7fb8f0362440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-15T04:28:06.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-15T04:28:06.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:28:06.110 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:28:06.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:28:06.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:28:06.592 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:28:06.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:28:06.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:28:06.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:28:06.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:28:06.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:28:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:06.612+0000 7f5c34f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:06.614 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:06.614+0000 7f5c34f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:06.616 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:06.615+0000 7f5c34f2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:28:06.980 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:28:06.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:06.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:07.129 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:07.129+0000 7f5c34f2d440 -1 Falling back to public interface 2024-09-15T04:28:07.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:08.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:08.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:08.336 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:28:08.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:28:08.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:08.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:08.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:09.625 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:09.624+0000 7f5c34f2d440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:28:09.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:09.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:09.692 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:28:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:28:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:10.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:11.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:11.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:11.322 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:28:11.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:28:11.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:11.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:11.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:12.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:12.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:12.747 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:28:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:28:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:14.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:14.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:14.160 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:28:14.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:28:14.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:14.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:14.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:15.633 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:28:15.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:28:15.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:15.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:16.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:17.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:17.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:17.029 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:28:17.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:28:17.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:17.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:17.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:18.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:18.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:18.491 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:28:18.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:28:18.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:18.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:19.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:20.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:20.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:20.009 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:28:20.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:28:20.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:20.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:20.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:21.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:21.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:21.465 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:28:21.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:28:21.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:21.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:21.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:22.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:22.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:22.947 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:28:22.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:28:22.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:22.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:23.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:24.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:24.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:24.897 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:28:24.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:28:24.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:24.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:25.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:26.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:26.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:26.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:28:26.643 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:28:26.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:26.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:27.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:28.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:28.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:28.438 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:28:28.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:28:28.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:28.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:29.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:30.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:30.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:30.529 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:28:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:28:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:30.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:31.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:32.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:32.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:32.622 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:28:32.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:28:32.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:32.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:28:33.076 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/198064,v1:127.0.0.1:6811/198064] [v2:127.0.0.1:6812/198064,v1:127.0.0.1:6813/198064] exists,up 5c01a9e6-3d18-4509-a1a1-3676688ec050 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:28:33.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:28:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:28:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:28:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-15T04:28:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-15T04:28:33.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:28:33.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:28:33.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=af308142-d72e-418f-b454-65a49cc3dc58 2024-09-15T04:28:33.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 af308142-d72e-418f-b454-65a49cc3dc58' 2024-09-15T04:28:33.086 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 af308142-d72e-418f-b454-65a49cc3dc58 2024-09-15T04:28:33.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:28:33.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxYuZmHz57BRAAIADaLiIYaSaSdbR+EtzViQ== 2024-09-15T04:28:33.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxYuZmHz57BRAAIADaLiIYaSaSdbR+EtzViQ=="}' 2024-09-15T04:28:33.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new af308142-d72e-418f-b454-65a49cc3dc58 -i td/osd-markdown/2/new.json 2024-09-15T04:28:33.552 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:28:33.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-15T04:28:33.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxYuZmHz57BRAAIADaLiIYaSaSdbR+EtzViQ== --osd-uuid af308142-d72e-418f-b454-65a49cc3dc58 2024-09-15T04:28:33.587 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.586+0000 7f2ec180b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.588+0000 7f2ec180b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.589+0000 7f2ec180b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.589+0000 7f2ec180b440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:28:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.590+0000 7f2ec180b440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:28:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.590+0000 7f2ec180b440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:28:33.668 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:33.667+0000 7f2ec180b440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-15T04:28:40.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-15T04:28:40.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:28:40.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:28:40.139 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:28:40.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:28:40.622 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:28:40.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:28:40.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:28:40.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:28:40.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:28:40.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:28:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:40.641+0000 7fba4d11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:40.643+0000 7fba4d11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:40.644+0000 7fba4d11d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:28:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:40.975 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:28:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:28:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:40.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:41.163 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:41.161+0000 7fba4d11d440 -1 Falling back to public interface 2024-09-15T04:28:41.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:42.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:42.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:42.334 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:28:42.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:28:42.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:42.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:42.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:43.644 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:28:43.644+0000 7fba4d11d440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:28:43.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:43.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:43.694 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:28:43.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:28:43.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:43.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:44.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:45.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:45.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:45.310 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:28:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:28:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:45.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:46.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:46.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:46.716 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:28:46.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:28:46.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:46.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:47.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:48.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:48.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:48.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:28:48.174 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:28:48.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:48.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:48.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:49.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:49.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:49.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:28:49.713 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:28:49.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:49.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:50.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:51.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:51.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:51.072 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:28:51.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:28:51.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:51.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:51.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:52.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:52.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:52.559 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:28:52.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:28:52.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:52.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:54.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:54.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:54.027 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:28:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:28:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:54.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:55.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:55.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:55.608 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:28:55.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:28:55.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:55.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:57.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:57.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:57.181 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:28:57.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:28:57.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:57.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:57.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:28:58.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:28:58.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:28:58.959 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:28:58.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:28:58.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:28:58.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:28:59.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:29:00.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:29:00.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:29:00.703 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:29:00.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:29:00.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:29:00.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:29:01.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:29:02.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:29:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:29:02.490 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:29:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:29:02.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:29:02.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:29:03.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:29:04.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:29:04.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:29:04.035 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:29:04.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:29:04.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:29:04.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:29:04.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:29:05.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:29:05.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:29:05.592 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:29:05.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:29:05.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:29:05.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:29:06.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:29:07.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:29:07.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:29:07.509 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:29:07.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:29:07.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:29:07.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:29:07.855 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/199794,v1:127.0.0.1:6819/199794] [v2:127.0.0.1:6820/199794,v1:127.0.0.1:6821/199794] exists,up af308142-d72e-418f-b454-65a49cc3dc58 2024-09-15T04:29:07.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:29:07.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:29:07.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:29:07.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-09-15T04:29:07.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T04:29:08.198 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T04:29:08.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T04:29:08.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T04:29:08.932 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T04:29:08.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:29:09.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T04:29:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-09-15T04:29:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-09-15T04:29:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-09-15T04:29:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-15T04:29:11.884 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:29:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-09-15T04:29:11.989 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:29:11.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-09-15T04:29:11.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-15T04:29:11.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-09-15T04:29:11.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-15T04:29:11.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-15T04:29:11.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:29:11.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:29:12.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:29:12.457 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:29:12.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:29:12.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:29:12.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:29:12.823 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:12.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:29:12.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:29:13.214 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:29:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:29:23.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:29:23.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:29:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:29:23.714 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:29:23.714 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:29:23.715 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:29:23.715 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:23.715 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:29:23.715 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:29:23.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:29:23.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:29:23.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:29:24.088 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:24.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:29:24.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:29:24.551 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:29:24.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:29:34.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:29:34.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:29:34.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:29:35.057 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:29:35.058 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:29:35.058 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:29:35.058 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:35.058 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:29:35.058 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:29:35.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:29:35.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:29:35.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:29:35.433 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:29:35.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:29:35.885 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:29:35.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:29:45.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:29:45.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:29:46.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:29:46.374 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:29:46.374 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:29:46.374 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:29:46.374 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:46.374 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:29:46.375 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:29:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:29:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:29:46.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:29:46.738 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:29:46.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:29:46.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:29:47.111 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:29:47.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:29:57.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-09-15T04:30:12.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:30:12.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-09-15T04:30:12.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-09-15T04:30:12.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-09-15T04:30:12.609 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:30:12.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-15T04:30:12.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:30:12.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:30:12.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:30:12.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:30:12.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:30:12.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:30:12.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:30:12.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:30:12.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:30:12.717 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:30:12.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:30:12.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:30:12.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:30:12.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:30:12.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:30:12.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:30:12.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:30:12.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:30:12.721 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:30:12.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:30:12.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:30:12.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:30:12.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:30:12.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:12.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:12.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-15T04:30:12.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:30:12.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:30:12.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:30:12.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:30:12.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:30:12.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:30:12.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:30:12.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:30:12.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:30:12.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:30:12.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:30:12.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:30:12.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:30:12.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:30:12.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:30:12.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:30:12.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:12.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:30:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:30:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:30:12.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-15T04:30:12.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:30:12.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:12.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:12.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:30:12.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-15T04:30:12.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-15T04:30:13.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:30:13.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:30:13.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:30:13.119 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:30:13.119 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:13.120 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:13.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:13.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:30:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:30:13.354 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:30:13.354 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:30:13.354 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:30:13.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:30:13.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:30:13.355 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:30:13.355 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:30:13.355 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:30:13.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T04:30:13.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:30:13.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:30:13.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-15T04:30:13.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:14.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:30:14.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:30:14.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:30:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:30:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:30:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c979f87e-3109-475d-9bc8-d451268a6abb 2024-09-15T04:30:14.362 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 c979f87e-3109-475d-9bc8-d451268a6abb 2024-09-15T04:30:14.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c979f87e-3109-475d-9bc8-d451268a6abb' 2024-09-15T04:30:14.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:30:14.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDWYuZmfe4ZFhAAD5wcJYlQL4pb1XVbn2Jdpw== 2024-09-15T04:30:14.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDWYuZmfe4ZFhAAD5wcJYlQL4pb1XVbn2Jdpw=="}' 2024-09-15T04:30:14.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c979f87e-3109-475d-9bc8-d451268a6abb -i td/osd-markdown/0/new.json 2024-09-15T04:30:14.677 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:30:14.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-15T04:30:14.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDWYuZmfe4ZFhAAD5wcJYlQL4pb1XVbn2Jdpw== --osd-uuid c979f87e-3109-475d-9bc8-d451268a6abb 2024-09-15T04:30:14.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.708+0000 7fde88830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:14.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.721+0000 7fde88830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:14.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.722+0000 7fde88830440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:14.724 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.722+0000 7fde88830440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:30:14.724 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.722+0000 7fde88830440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:30:14.724 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.722+0000 7fde88830440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:30:14.754 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:14.752+0000 7fde88830440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-15T04:30:20.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-15T04:30:20.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:30:20.337 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:30:20.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:30:20.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:30:20.815 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:30:20.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:30:20.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:30:20.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:30:20.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:30:20.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:30:20.835 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:20.835+0000 7f4ce5921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:20.837 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:20.837+0000 7f4ce5921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:20.838 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:20.838+0000 7f4ce5921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:21.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:30:21.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:30:21.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:30:21.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:30:21.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:30:21.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:21.162 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:30:21.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:30:21.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:21.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:21.355 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:21.355+0000 7f4ce5921440 -1 Falling back to public interface 2024-09-15T04:30:21.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:22.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:22.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:22.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:30:22.524 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:30:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:22.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:23.855 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:23.854+0000 7f4ce5921440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:30:23.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:23.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:23.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:30:23.880 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:30:23.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:23.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:24.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:25.235 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:30:25.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:30:25.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:25.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:25.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:26.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:26.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:26.693 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:30:26.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:30:26.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:26.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:27.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:28.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:28.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:28.051 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:30:28.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:30:28.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:28.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:28.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:29.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:29.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:29.547 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:30:29.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:30:29.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:29.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:29.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:30.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:30.926 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:30:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:30:30.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:30.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:31.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:32.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:32.306 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:30:32.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:32.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:30:32.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:32.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:32.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:33.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:33.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:33.687 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:30:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:30:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:34.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:35.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:35.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:35.101 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:30:35.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:30:35.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:35.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:35.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:36.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:36.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:36.545 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:30:36.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:30:36.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:36.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:36.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:37.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:37.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:37.934 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:30:37.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:30:37.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:37.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:38.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:39.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:39.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:39.709 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:30:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:30:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:40.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:41.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:41.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:41.301 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:30:41.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:30:41.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:41.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:42.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:43.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:43.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:43.016 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:30:43.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:30:43.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:43.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:43.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:44.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:44.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:44.833 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:30:44.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:30:44.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:44.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:45.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:46.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:46.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:46.606 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:30:46.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:30:46.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:46.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:30:46.979 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/202367,v1:127.0.0.1:6803/202367] [v2:127.0.0.1:6804/202367,v1:127.0.0.1:6805/202367] exists,up c979f87e-3109-475d-9bc8-d451268a6abb 2024-09-15T04:30:46.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:30:46.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:30:46.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-15T04:30:46.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:30:46.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=098dcec2-6a63-4f26-9662-29ebaeb94286 2024-09-15T04:30:46.987 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 098dcec2-6a63-4f26-9662-29ebaeb94286 2024-09-15T04:30:46.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 098dcec2-6a63-4f26-9662-29ebaeb94286' 2024-09-15T04:30:46.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:30:46.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD2YuZmd71ZOxAAFFgkhlF0kN+sotP/SuIeow== 2024-09-15T04:30:46.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD2YuZmd71ZOxAAFFgkhlF0kN+sotP/SuIeow=="}' 2024-09-15T04:30:46.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 098dcec2-6a63-4f26-9662-29ebaeb94286 -i td/osd-markdown/1/new.json 2024-09-15T04:30:47.465 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:30:47.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-15T04:30:47.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD2YuZmd71ZOxAAFFgkhlF0kN+sotP/SuIeow== --osd-uuid 098dcec2-6a63-4f26-9662-29ebaeb94286 2024-09-15T04:30:47.498 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.498+0000 7f73da80b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:47.500 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.500+0000 7f73da80b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:47.501 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.501+0000 7f73da80b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:47.502 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.501+0000 7f73da80b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:30:47.502 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.501+0000 7f73da80b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:30:47.502 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.501+0000 7f73da80b440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:30:47.546 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:47.546+0000 7f73da80b440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-15T04:30:53.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-15T04:30:53.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:30:53.479 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:30:53.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:30:53.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:30:53.971 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:30:53.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:30:53.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:30:53.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:30:53.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:30:53.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:30:53.992 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:53.991+0000 7fcff0e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:54.019 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:54.018+0000 7fcff0e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:54.020 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:54.019+0000 7fcff0e69440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:30:54.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:54.320 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:30:54.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:30:54.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:54.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:30:54.553 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:54.553+0000 7fcff0e69440 -1 Falling back to public interface 2024-09-15T04:30:54.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:55.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:55.670 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:30:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:30:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:30:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:57.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:57.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:57.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:30:57.024 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:30:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:30:57.139 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:30:57.137+0000 7fcff0e69440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:30:57.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:58.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:58.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:58.440 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:30:58.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:30:58.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:58.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:30:58.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:30:59.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:30:59.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:30:59.990 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:30:59.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:30:59.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:30:59.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:01.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:01.379 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:31:01.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:31:01.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:01.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:01.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:02.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:02.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:02.979 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:31:02.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:31:02.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:02.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:03.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:04.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:04.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:04.341 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:31:04.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:31:04.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:04.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:04.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:05.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:05.870 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:31:05.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:05.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:31:05.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:05.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:06.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:07.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:07.319 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:31:07.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:31:07.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:07.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:07.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:08.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:08.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:08.723 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:31:08.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:31:08.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:08.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:09.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:10.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:10.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:10.088 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:31:10.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:31:10.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:10.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:10.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:11.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:11.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:11.839 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:31:11.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:31:11.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:11.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:12.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:13.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:13.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:13.643 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:31:13.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:31:13.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:13.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:14.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:15.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:15.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:15.380 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:31:15.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:31:15.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:15.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:16.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:17.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:17.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:17.081 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:31:17.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:31:17.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:17.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:17.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:18.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:18.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:18.843 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:31:18.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:31:18.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:18.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:19.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:20.517 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:31:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:31:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:31:20.878 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/204121,v1:127.0.0.1:6811/204121] [v2:127.0.0.1:6812/204121,v1:127.0.0.1:6813/204121] exists,up 098dcec2-6a63-4f26-9662-29ebaeb94286 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:31:20.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:31:20.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:31:20.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89a39c4c-e383-4052-92c6-d6e84af5db25 2024-09-15T04:31:20.885 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 89a39c4c-e383-4052-92c6-d6e84af5db25 2024-09-15T04:31:20.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 89a39c4c-e383-4052-92c6-d6e84af5db25' 2024-09-15T04:31:20.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:31:20.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAYY+ZmRstFNRAA8lwR6Z31sizM+nNAVWokAQ== 2024-09-15T04:31:20.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAYY+ZmRstFNRAA8lwR6Z31sizM+nNAVWokAQ=="}' 2024-09-15T04:31:20.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89a39c4c-e383-4052-92c6-d6e84af5db25 -i td/osd-markdown/2/new.json 2024-09-15T04:31:21.470 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:31:21.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-15T04:31:21.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAYY+ZmRstFNRAA8lwR6Z31sizM+nNAVWokAQ== --osd-uuid 89a39c4c-e383-4052-92c6-d6e84af5db25 2024-09-15T04:31:21.504 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.503+0000 7f568511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:21.505 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.505+0000 7f568511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:21.506 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.506+0000 7f568511f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:21.507 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.506+0000 7f568511f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:31:21.507 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.506+0000 7f568511f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:31:21.507 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.506+0000 7f568511f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:31:21.552 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:21.551+0000 7f568511f440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-15T04:31:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-15T04:31:27.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:31:27.690 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:31:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:31:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:31:28.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:31:28.181 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:31:28.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:31:28.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:31:28.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:31:28.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:31:28.202 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:28.201+0000 7ff6e7b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:28.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:28.203+0000 7ff6e7b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:28.204 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:28.204+0000 7ff6e7b21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:31:28.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:28.528 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:31:28.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:31:28.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:28.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:28.716 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:28.716+0000 7ff6e7b21440 -1 Falling back to public interface 2024-09-15T04:31:28.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:29.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:29.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:29.906 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:31:29.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:31:29.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:29.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:30.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:31.199 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:31:31.198+0000 7ff6e7b21440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:31:31.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:31.255 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:31:31.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:31.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:31:31.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:31.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:31.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:32.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:32.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:32.869 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:31:32.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:31:32.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:32.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:33.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:34.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:34.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:34.351 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:31:34.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:31:34.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:34.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:34.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:35.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:35.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:35.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:31:35.826 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:31:35.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:35.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:36.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:37.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:37.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:37.230 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:31:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:31:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:37.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:38.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:38.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:38.593 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:31:38.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:31:38.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:38.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:40.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:40.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:40.110 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:31:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:31:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:40.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:40.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:41.501 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:31:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:31:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:41.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:42.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:42.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:42.991 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:31:42.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:31:42.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:42.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:43.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:44.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:44.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:44.379 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:31:44.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:31:44.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:44.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:45.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:46.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:46.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:46.334 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:31:46.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:31:46.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:46.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:47.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:48.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:48.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:48.111 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:31:48.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:31:48.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:48.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:48.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:49.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:49.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:49.955 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:31:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:31:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:49.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:50.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:51.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:51.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:51.440 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:31:51.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:31:51.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:51.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:52.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:53.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:53.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:53.185 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:31:53.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:31:53.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:53.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:31:54.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:31:54.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:31:54.964 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:31:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:31:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:31:54.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:31:55.390 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/205875,v1:127.0.0.1:6819/205875] [v2:127.0.0.1:6820/205875,v1:127.0.0.1:6821/205875] exists,up 89a39c4c-e383-4052-92c6-d6e84af5db25 2024-09-15T04:31:55.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:31:55.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:31:55.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:31:55.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-09-15T04:31:55.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T04:31:55.738 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T04:31:55.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T04:31:55.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T04:31:56.513 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T04:31:56.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:31:57.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T04:31:59.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-09-15T04:31:59.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-09-15T04:31:59.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-09-15T04:31:59.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-09-15T04:31:59.465 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:31:59.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-09-15T04:31:59.571 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:31:59.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-09-15T04:31:59.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-09-15T04:31:59.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-09-15T04:31:59.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-09-15T04:31:59.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-09-15T04:31:59.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:31:59.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:31:59.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:32:00.016 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:32:00.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:32:00.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:32:00.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:32:00.395 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:00.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:32:00.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:32:00.768 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:32:00.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:32:10.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:32:10.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:32:10.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:32:11.229 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:32:11.230 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:32:11.230 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:32:11.230 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:11.230 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:32:11.230 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:32:11.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:32:11.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:32:11.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:32:11.645 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:11.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:32:11.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:32:12.135 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:32:12.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:32:22.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:32:22.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:32:22.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:32:22.630 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:32:22.630 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:32:22.630 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:32:22.630 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:22.630 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:32:22.631 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:32:22.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:32:22.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:32:22.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:32:23.019 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:23.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:32:23.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:32:23.418 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:32:23.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:32:33.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-09-15T04:32:33.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-09-15T04:32:33.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-09-15T04:32:33.879 INFO:tasks.workunit.client.0.smithi138.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-09-15T04:32:33.880 INFO:tasks.workunit.client.0.smithi138.stdout:-1 0.29306 root default 2024-09-15T04:32:33.880 INFO:tasks.workunit.client.0.smithi138.stdout:-3 0.29306 host smithi138 2024-09-15T04:32:33.880 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:33.880 INFO:tasks.workunit.client.0.smithi138.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-09-15T04:32:33.880 INFO:tasks.workunit.client.0.smithi138.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-09-15T04:32:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-09-15T04:32:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-09-15T04:32:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-09-15T04:32:34.242 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:32:34.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-09-15T04:32:34.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-09-15T04:32:34.620 INFO:tasks.workunit.client.0.smithi138.stderr:marked down osd.0. 2024-09-15T04:32:34.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-09-15T04:32:44.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-09-15T04:32:44.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-09-15T04:32:44.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-09-15T04:32:44.990 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:32:44.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:32:44.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:32:44.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:32:45.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:32:45.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:32:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:32:45.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:32:45.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:32:45.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:32:45.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:32:45.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:32:45.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:32:45.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:32:45.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:32:45.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:32:45.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:32:45.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:32:45.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:32:45.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:32:45.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:32:45.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:32:45.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:32:45.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:32:45.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:32:45.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:32:45.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:32:45.123 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:32:45.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:32:45.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:32:45.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:32:45.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:32:45.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:32:45.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:32:45.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:32:45.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-09-15T04:32:45.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:32:45.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.189207 2024-09-15T04:32:45.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:32:45.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:32:45.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:32:45.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-09-15T04:32:45.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:32:45.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:32:45.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:32:45.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:32:45.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:32:45.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:32:45.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:32:45.728 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.729 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:32:45.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:32:45.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get fsid 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:32:45.800 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.189207/ceph-mon.a.asok 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:32:45.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.189207/ceph-mon.a.asok config get mon_host 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-09-15T04:32:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:32:46.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:32:46.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:32:46.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:32:46.702 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:32:46.702 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:46.703 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:46.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:32:46.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:32:46.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:32:46.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-09-15T04:32:46.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:32:46.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:32:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-09-15T04:32:46.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:32:46.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=098320ac-fb95-4bbf-9b96-45acab92bfbf 2024-09-15T04:32:46.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 098320ac-fb95-4bbf-9b96-45acab92bfbf' 2024-09-15T04:32:46.731 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 098320ac-fb95-4bbf-9b96-45acab92bfbf 2024-09-15T04:32:46.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:32:46.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBuY+Zm46YiLBAAbfp8sBXAyhz5DKN7wbhYIQ== 2024-09-15T04:32:46.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBuY+Zm46YiLBAAbfp8sBXAyhz5DKN7wbhYIQ=="}' 2024-09-15T04:32:46.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 098320ac-fb95-4bbf-9b96-45acab92bfbf -i td/osd-markdown/0/new.json 2024-09-15T04:32:47.079 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:32:47.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-09-15T04:32:47.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBuY+Zm46YiLBAAbfp8sBXAyhz5DKN7wbhYIQ== --osd-uuid 098320ac-fb95-4bbf-9b96-45acab92bfbf 2024-09-15T04:32:47.113 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.112+0000 7f22fb40d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:47.125 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.125+0000 7f22fb40d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:47.126 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.126+0000 7f22fb40d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:47.127 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.126+0000 7f22fb40d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:32:47.127 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.126+0000 7f22fb40d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:32:47.127 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.126+0000 7f22fb40d440 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-09-15T04:32:47.166 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:47.165+0000 7f22fb40d440 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-09-15T04:32:52.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-09-15T04:32:52.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:32:52.490 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:32:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:32:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:32:52.978 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:32:52.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:32:52.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:32:52.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:32:52.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:32:52.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:32:52.999 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:52.998+0000 7fcdbca15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:53.000 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:53.000+0000 7fcdbca15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:53.001 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:53.001+0000 7fcdbca15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:32:53.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:32:53.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:32:53.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:32:53.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:32:53.520 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:53.520+0000 7fcdbca15440 -1 Falling back to public interface 2024-09-15T04:32:53.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:32:54.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:32:54.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:32:54.673 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:32:54.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:32:54.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:32:54.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:32:55.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:32:55.995 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:32:55.994+0000 7fcdbca15440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:32:56.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:32:56.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:32:56.031 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:32:56.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:32:56.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:32:56.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:32:56.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:32:57.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:32:57.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:32:57.385 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:32:57.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:32:57.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:32:57.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:32:57.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:32:58.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:32:58.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:32:58.859 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:32:58.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:32:58.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:32:58.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:32:59.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:00.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:00.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:00.251 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:33:00.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:33:00.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:00.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:00.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:01.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:01.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:01.681 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:33:01.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:33:01.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:01.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:02.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:03.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:03.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:03.082 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:33:03.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:33:03.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:03.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:03.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:04.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:04.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:04.457 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:33:04.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:33:04.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:04.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:04.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:05.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:05.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:05.844 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:33:05.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:33:05.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:05.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:06.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:07.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:07.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:07.230 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:33:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:33:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:07.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:08.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:08.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:08.643 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:33:08.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:33:08.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:08.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:09.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:10.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:10.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:10.277 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:33:10.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:33:10.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:10.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:11.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:12.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:12.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:12.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:33:12.439 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:33:12.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:12.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:13.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:14.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:14.285 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:33:14.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:14.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:33:14.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:14.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:15.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:16.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:16.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:16.108 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:33:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:33:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:16.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:17.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:17.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:17.897 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:33:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:33:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:17.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:18.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:19.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:19.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:19.758 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:33:19.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:33:19.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:19.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:33:20.112 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/208422,v1:127.0.0.1:6803/208422] [v2:127.0.0.1:6804/208422,v1:127.0.0.1:6805/208422] exists,up 098320ac-fb95-4bbf-9b96-45acab92bfbf 2024-09-15T04:33:20.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:33:20.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:33:20.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-09-15T04:33:20.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:33:20.119 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 e2d1c0e0-d25d-452e-8e83-f7cd61ce2326 2024-09-15T04:33:20.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2d1c0e0-d25d-452e-8e83-f7cd61ce2326 2024-09-15T04:33:20.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e2d1c0e0-d25d-452e-8e83-f7cd61ce2326' 2024-09-15T04:33:20.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:33:20.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQY+Zmc9ajBxAALMVX6lY2TGEPYgpqnA64YA== 2024-09-15T04:33:20.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQY+Zmc9ajBxAALMVX6lY2TGEPYgpqnA64YA=="}' 2024-09-15T04:33:20.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2d1c0e0-d25d-452e-8e83-f7cd61ce2326 -i td/osd-markdown/1/new.json 2024-09-15T04:33:20.645 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:33:20.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-09-15T04:33:20.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQY+Zmc9ajBxAALMVX6lY2TGEPYgpqnA64YA== --osd-uuid e2d1c0e0-d25d-452e-8e83-f7cd61ce2326 2024-09-15T04:33:20.679 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.677+0000 7f0368123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:20.681 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.679+0000 7f0368123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:20.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.680+0000 7f0368123440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:20.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.680+0000 7f0368123440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:33:20.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.680+0000 7f0368123440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:33:20.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.680+0000 7f0368123440 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-09-15T04:33:20.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:20.726+0000 7f0368123440 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-09-15T04:33:26.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-09-15T04:33:26.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:33:26.609 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:33:26.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:33:26.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:33:27.088 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:33:27.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:33:27.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:33:27.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:33:27.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:33:27.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:33:27.109 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:27.109+0000 7f15a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:27.135 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:27.135+0000 7f15a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:27.136 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:27.136+0000 7f15a0521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:33:27.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:27.481 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:33:27.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:33:27.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:27.668 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:27.667+0000 7f15a0521440 -1 Falling back to public interface 2024-09-15T04:33:27.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:28.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:28.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:28.830 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:33:28.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:33:28.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:28.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:29.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:30.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:30.155+0000 7f15a0521440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:33:30.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:30.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:30.190 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:33:30.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:33:30.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:30.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:30.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:31.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:31.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:31.836 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:33:31.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:33:31.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:31.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:33.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:33.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:33.242 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:33:33.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:33:33.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:33.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:33.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:34.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:34.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:34.628 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:33:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:33:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:35.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:36.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:36.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:36.044 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:33:36.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:33:36.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:36.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:36.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:37.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:37.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:37.492 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:33:37.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:33:37.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:37.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:37.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:38.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:38.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:38.900 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:33:38.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:33:38.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:38.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:39.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:40.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:40.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:40.359 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:33:40.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:33:40.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:40.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:40.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:41.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:41.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:41.745 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:33:41.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:33:41.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:41.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:42.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:43.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:43.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:43.244 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:33:43.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:33:43.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:43.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:44.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:45.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:45.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:45.325 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:33:45.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:33:45.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:45.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:46.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:47.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:47.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:47.118 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:33:47.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:33:47.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:47.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:47.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:48.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:48.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:48.897 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:33:48.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:33:48.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:48.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:49.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:50.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:50.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:50.371 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:33:50.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:33:50.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:50.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:51.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:52.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:52.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:52.568 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:33:52.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:33:52.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:52.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:53.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:33:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:33:54.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:33:54.028 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:33:54.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:33:54.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:33:54.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:33:54.421 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/210176,v1:127.0.0.1:6811/210176] [v2:127.0.0.1:6812/210176,v1:127.0.0.1:6813/210176] exists,up e2d1c0e0-d25d-452e-8e83-f7cd61ce2326 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:33:54.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:33:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:33:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=138d8819-9b39-42c2-90db-916b0cdaf510 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 138d8819-9b39-42c2-90db-916b0cdaf510 2024-09-15T04:33:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 138d8819-9b39-42c2-90db-916b0cdaf510' 2024-09-15T04:33:54.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:33:54.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCyY+Zm/dUIGhAAtw1vvAce64t6xJJLYJnIgA== 2024-09-15T04:33:54.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCyY+Zm/dUIGhAAtw1vvAce64t6xJJLYJnIgA=="}' 2024-09-15T04:33:54.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 138d8819-9b39-42c2-90db-916b0cdaf510 -i td/osd-markdown/2/new.json 2024-09-15T04:33:54.915 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:33:54.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-09-15T04:33:54.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCyY+Zm/dUIGhAAtw1vvAce64t6xJJLYJnIgA== --osd-uuid 138d8819-9b39-42c2-90db-916b0cdaf510 2024-09-15T04:33:54.949 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.948+0000 7f32f260f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:54.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.950+0000 7f32f260f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:54.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.951+0000 7f32f260f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:33:54.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.951+0000 7f32f260f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:33:54.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.951+0000 7f32f260f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:33:54.954 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.951+0000 7f32f260f440 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-09-15T04:33:54.997 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:33:54.997+0000 7f32f260f440 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-09-15T04:34:00.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-09-15T04:34:00.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:34:00.897 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:34:00.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:34:00.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:34:01.375 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:34:01.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:34:01.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a72744b9-4eb2-47e0-b4c0-c24ec8853f05 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.189207/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:34:01.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:34:01.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:34:01.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:34:01.395 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:01.395+0000 7f78abb23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:01.398 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:01.397+0000 7f78abb23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:01.398 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:01.398+0000 7f78abb23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:34:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:34:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:34:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:34:01.761 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:34:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:34:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:34:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:01.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:01.918 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:01.918+0000 7f78abb23440 -1 Falling back to public interface 2024-09-15T04:34:02.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:03.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:03.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:03.155 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:34:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:34:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:03.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:03.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:04.478 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:04.477+0000 7f78abb23440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:34:04.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:04.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:04.502 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:34:04.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:34:04.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:04.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:04.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:05.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:05.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:05.932 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:34:05.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:34:05.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:05.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:06.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:07.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:07.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:07.461 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:34:07.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:34:07.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:07.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:07.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:08.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:08.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:08.842 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:34:08.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:34:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:09.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:10.388 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:34:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:34:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:10.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:11.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:11.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:11.782 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:34:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:34:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:12.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:13.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:13.224 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:34:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:34:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:13.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:13.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:14.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:14.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:14.720 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:34:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:34:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:16.185 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:34:16.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:34:16.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:16.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:16.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:17.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:17.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:17.597 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:34:17.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:34:17.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:17.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:18.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:19.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:19.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:19.268 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:34:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:34:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:19.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:20.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:21.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:21.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:21.021 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:34:21.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:34:21.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:21.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:21.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:22.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:22.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:22.865 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:34:22.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:34:22.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:22.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:24.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:25.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:25.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:25.070 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:34:25.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:34:25.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:25.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:25.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:26.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:26.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:26.537 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:34:26.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:34:26.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:26.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:27.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:28.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:28.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:28.458 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:34:28.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:34:28.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:28.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:34:28.851 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/211930,v1:127.0.0.1:6819/211930] [v2:127.0.0.1:6820/211930,v1:127.0.0.1:6821/211930] exists,up 138d8819-9b39-42c2-90db-916b0cdaf510 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=208422 2024-09-15T04:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 208422 2024-09-15T04:34:28.858 INFO:tasks.workunit.client.0.smithi138.stdout: PID TTY TIME CMD 2024-09-15T04:34:28.858 INFO:tasks.workunit.client.0.smithi138.stdout: 208422 ? 00:00:00 ceph-osd 2024-09-15T04:34:28.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-09-15T04:34:28.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-09-15T04:34:28.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-09-15T04:34:29.213 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-09-15T04:34:29.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-09-15T04:34:29.721 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already stopped. 2024-09-15T04:34:29.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-09-15T04:34:44.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-09-15T04:34:44.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-09-15T04:34:44.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-09-15T04:34:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-09-15T04:34:45.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 208422 2024-09-15T04:34:45.097 INFO:tasks.workunit.client.0.smithi138.stdout: PID TTY TIME CMD 2024-09-15T04:34:45.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:34:45.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:34:45.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:34:45.206 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:34:45.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:34:45.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:34:45.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:34:45.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:34:45.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:34:45.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:34:45.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:34:45.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:34:45.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:34:45.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:34:45.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:34:45.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:34:45.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:34:45.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:34:45.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-markdown 0 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:34:45.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:34:45.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:34:45.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:34:45.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:34:45.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:34:45.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:34:45.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:34:45.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:34:45.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:34:45.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:34:45.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:34:45.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:34:45.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-09-15T04:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.189207 2024-09-15T04:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.189207 2024-09-15T04:34:45.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:34:45.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:34:45.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:34:45.235 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:34:45.235 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:34:45.299 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-09-15T04:34:45.300 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-09-15T04:34:45.361 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:34:45.366 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:34:45.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:34:45.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-reactivate 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-09-15T04:34:45.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-09-15T04:34:45.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-09-15T04:34:45.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none ' 2024-09-15T04:34:45.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-09-15T04:34:45.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:34:45.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:34:45.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:34:45.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:34:45.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:34:45.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:34:45.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:34:45.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:34:45.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:34:45.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:34:45.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:34:45.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:34:45.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:34:45.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:34:45.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:34:45.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:34:45.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-15T04:34:45.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:34:45.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:45.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.213006 2024-09-15T04:34:45.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:34:45.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:34:45.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-09-15T04:34:45.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:34:45.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:45.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.213006 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-09-15T04:34:45.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-09-15T04:34:45.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-09-15T04:34:45.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:34:45.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:34:45.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:34:45.725 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:34:45.726 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.726 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:45.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:34:45.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:34:45.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:34:45.979 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:45.980 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:45.980 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.213006/ceph-mon.a.asok 2024-09-15T04:34:45.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:34:45.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.213006/ceph-mon.a.asok config get fsid 2024-09-15T04:34:46.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:34:46.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:34:46.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.213006/ceph-mon.a.asok 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:34:46.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.213006/ceph-mon.a.asok config get mon_host 2024-09-15T04:34:46.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-09-15T04:34:46.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-09-15T04:34:46.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:34:46.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:34:46.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:34:46.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-09-15T04:34:46.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:34:46.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:34:46.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:34:47.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-09-15T04:34:47.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-09-15T04:34:47.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:34:47.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:34:47.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-15T04:34:47.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:34:47.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4090c175-89a5-447d-97cd-58323309b937 2024-09-15T04:34:47.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4090c175-89a5-447d-97cd-58323309b937' 2024-09-15T04:34:47.004 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 4090c175-89a5-447d-97cd-58323309b937 2024-09-15T04:34:47.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:34:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnY+ZmLMqxABAA+CyXxuWgMd8qhVF+ipvkBg== 2024-09-15T04:34:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnY+ZmLMqxABAA+CyXxuWgMd8qhVF+ipvkBg=="}' 2024-09-15T04:34:47.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4090c175-89a5-447d-97cd-58323309b937 -i td/osd-reactivate/0/new.json 2024-09-15T04:34:47.341 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:34:47.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-09-15T04:34:47.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnY+ZmLMqxABAA+CyXxuWgMd8qhVF+ipvkBg== --osd-uuid 4090c175-89a5-447d-97cd-58323309b937 2024-09-15T04:34:47.374 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.373+0000 7f472b921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:47.386 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.386+0000 7f472b921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:47.388 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.387+0000 7f472b921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:47.388 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.387+0000 7f472b921440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-15T04:34:47.388 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.387+0000 7f472b921440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-15T04:34:47.388 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.387+0000 7f472b921440 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-09-15T04:34:47.426 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:47.426+0000 7f472b921440 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-09-15T04:34:52.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-09-15T04:34:52.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:34:52.747 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:34:52.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:34:52.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:34:53.224 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:34:53.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:34:53.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:34:53.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:34:53.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:34:53.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:34:53.245 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:53.243+0000 7fc4e000c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:53.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:53.245+0000 7fc4e000c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:53.248 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:53.246+0000 7fc4e000c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:34:53.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:34:53.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:34:53.572 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:34:53.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:53.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:34:53.761 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:53.760+0000 7fc4e000c440 -1 Falling back to public interface 2024-09-15T04:34:53.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:54.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:54.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:54.929 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:34:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:34:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:34:55.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:56.219 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:34:56.217+0000 7fc4e000c440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:34:56.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:56.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:56.281 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:34:56.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:34:56.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:56.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:34:56.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:57.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:57.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:57.772 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:34:57.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:34:57.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:57.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:34:58.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:34:59.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:34:59.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:34:59.153 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:34:59.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:34:59.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:34:59.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:34:59.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:00.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:00.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:00.521 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:35:00.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:35:00.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:00.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:00.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:01.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:01.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:01.913 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:35:01.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:35:01.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:01.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:02.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:03.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:03.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:03.299 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:35:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:35:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:03.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:03.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:04.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:04.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:04.688 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:35:04.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:35:04.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:04.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:05.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:06.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:06.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:06.177 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:35:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:35:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:06.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:07.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:07.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:07.574 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:35:07.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:35:07.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:07.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:08.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:09.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:09.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:09.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:35:09.646 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:35:09.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:09.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:10.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:11.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:11.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:11.213 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:35:11.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:35:11.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:11.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:11.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:12.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:12.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:12.874 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:35:12.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:35:12.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:12.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:13.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:14.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:14.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:14.677 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:35:14.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:35:14.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:14.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:15.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:16.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:16.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:35:16.753 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:35:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:17.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:18.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:18.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:18.243 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:35:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:35:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:18.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:18.773 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/214097,v1:127.0.0.1:6803/214097] [v2:127.0.0.1:6804/214097,v1:127.0.0.1:6805/214097] exists,up 4090c175-89a5-447d-97cd-58323309b937 2024-09-15T04:35:18.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:35:18.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:35:18.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:35:19.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:35:19.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-09-15T04:35:19.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-09-15T04:35:19.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-09-15T04:35:19.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-09-15T04:35:19.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:35:19.086 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:19.087 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:35:19.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:35:19.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-09-15T04:35:19.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T04:35:19.089 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3c7695f6-2b1e-4400-adf4-7239b5971b6d --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.213006/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T04:35:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T04:35:19.104 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:19.103+0000 7f371521d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:19.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:19.143+0000 7f371521d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:19.145 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:19.144+0000 7f371521d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T04:35:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:35:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:35:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:19.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:19.659 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:19.658+0000 7f371521d440 -1 Falling back to public interface 2024-09-15T04:35:19.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:20.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:20.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:20.866 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:35:20.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:35:20.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:20.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:21.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:22.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:22.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:22.258 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:35:22.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:35:22.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:22.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:22.500 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:22.499+0000 7f371521d440 -1 osd.0 5 log_to_monitors true 2024-09-15T04:35:22.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:23.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:23.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:23.611 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:35:23.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:35:23.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:23.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:23.991 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/215011,v1:127.0.0.1:6803/215011] [v2:127.0.0.1:6804/215011,v1:127.0.0.1:6805/215011] exists,up 4090c175-89a5-447d-97cd-58323309b937 2024-09-15T04:35:23.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:35:23.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:35:23.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:35:23.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:35:24.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:35:24.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:35:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:35:24.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:35:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:35:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:35:24.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:35:24.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:35:24.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:35:24.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:35:24.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:35:24.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-15T04:35:24.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:35:24.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:35:24.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.213006 2024-09-15T04:35:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:35:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:35:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:35:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-reactivate 0 2024-09-15T04:35:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:35:24.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:35:24.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:35:24.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:35:24.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:35:24.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:35:24.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:35:24.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:35:24.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:35:24.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:35:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:35:24.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:35:24.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:35:24.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-09-15T04:35:24.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:35:24.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.213006 2024-09-15T04:35:24.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.213006 2024-09-15T04:35:24.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:35:24.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:35:24.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:35:24.121 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:35:24.121 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:35:24.186 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-09-15T04:35:24.186 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-09-15T04:35:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:35:24.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-prio 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none ' 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-09-15T04:35:24.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_priority' 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-15T04:35:24.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:35:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:35:24.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:35:24.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:35:24.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:35:24.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-15T04:35:24.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:35:24.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:24.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215650 2024-09-15T04:35:24.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:35:24.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:35:24.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-15T04:35:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:35:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:24.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.215650 2024-09-15T04:35:24.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:35:24.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:35:24.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:35:24.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-15T04:35:24.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:24.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:35:24.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:35:24.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:35:24.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:35:24.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.891 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:24.892 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-mon.a.asok 2024-09-15T04:35:24.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:35:24.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215650/ceph-mon.a.asok config get fsid 2024-09-15T04:35:24.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:35:24.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:35:24.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-mon.a.asok 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:35:24.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215650/ceph-mon.a.asok config get mon_host 2024-09-15T04:35:25.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-15T04:35:25.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:35:25.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:35:25.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:35:25.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:35:25.883 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:35:25.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:25.884 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:25.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:25.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:35:25.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:35:25.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-09-15T04:35:25.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-09-15T04:35:25.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-09-15T04:35:25.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:35:25.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-09-15T04:35:25.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:35:25.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-15T04:35:25.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:35:25.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=70c822f3-afd4-4aba-8732-12b6ca40e579 2024-09-15T04:35:25.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 70c822f3-afd4-4aba-8732-12b6ca40e579' 2024-09-15T04:35:25.918 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 70c822f3-afd4-4aba-8732-12b6ca40e579 2024-09-15T04:35:25.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:35:25.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANZOZmHW5cNxAAleeE9Iu7bGAAtgOsWq/BBA== 2024-09-15T04:35:25.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANZOZmHW5cNxAAleeE9Iu7bGAAtgOsWq/BBA=="}' 2024-09-15T04:35:25.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 70c822f3-afd4-4aba-8732-12b6ca40e579 -i td/osd-recovery-prio/0/new.json 2024-09-15T04:35:26.219 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:35:26.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-15T04:35:26.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANZOZmHW5cNxAAleeE9Iu7bGAAtgOsWq/BBA== --osd-uuid 70c822f3-afd4-4aba-8732-12b6ca40e579 2024-09-15T04:35:26.255 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.255+0000 7f770ae43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.264+0000 7f770ae43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:26.265 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.265+0000 7f770ae43440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:26.266 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.265+0000 7f770ae43440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:35:26.266 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.265+0000 7f770ae43440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:35:26.266 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.265+0000 7f770ae43440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:35:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:26.304+0000 7f770ae43440 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-15T04:35:31.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-15T04:35:31.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:35:31.676 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:35:31.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:35:31.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:35:32.157 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:35:32.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:35:32.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:35:32.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:35:32.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:35:32.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:35:32.182 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:32.181+0000 7f604f61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:32.184 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:32.183+0000 7f604f61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:32.185 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:32.184+0000 7f604f61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:32.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:35:32.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:35:32.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:35:32.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:35:32.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:35:32.508 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:35:32.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:32.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:35:32.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:32.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:32.698 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:32.698+0000 7f604f61d440 -1 Falling back to public interface 2024-09-15T04:35:32.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:33.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:33.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:33.909 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:35:33.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:35:33.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:33.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:34.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:35.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:35.189+0000 7f604f61d440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:35:35.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:35.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:35.270 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:35:35.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:35:35.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:35.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:35.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:36.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:36.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:36.664 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:35:36.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:35:36.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:36.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:35:37.014 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/216743,v1:127.0.0.1:6803/216743] [v2:127.0.0.1:6804/216743,v1:127.0.0.1:6805/216743] exists,up 70c822f3-afd4-4aba-8732-12b6ca40e579 2024-09-15T04:35:37.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:35:37.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:35:37.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:35:37.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:35:37.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:35:37.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:35:37.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-15T04:35:37.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:35:37.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4496238b-0c9d-447b-9e73-d3e113bd2666 2024-09-15T04:35:37.022 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 4496238b-0c9d-447b-9e73-d3e113bd2666 2024-09-15T04:35:37.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4496238b-0c9d-447b-9e73-d3e113bd2666' 2024-09-15T04:35:37.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:35:37.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZZOZmztf9ARAAdAT6bQRrKy+kPzIy4mARPQ== 2024-09-15T04:35:37.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZZOZmztf9ARAAdAT6bQRrKy+kPzIy4mARPQ=="}' 2024-09-15T04:35:37.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4496238b-0c9d-447b-9e73-d3e113bd2666 -i td/osd-recovery-prio/1/new.json 2024-09-15T04:35:37.533 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:35:37.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-15T04:35:37.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZZOZmztf9ARAAdAT6bQRrKy+kPzIy4mARPQ== --osd-uuid 4496238b-0c9d-447b-9e73-d3e113bd2666 2024-09-15T04:35:37.572 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.571+0000 7fb35735e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:37.592 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.592+0000 7fb35735e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:37.593 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.593+0000 7fb35735e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:37.594 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.593+0000 7fb35735e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:35:37.594 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.593+0000 7fb35735e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:35:37.594 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.593+0000 7fb35735e440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:35:37.684 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:37.683+0000 7fb35735e440 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-15T04:35:43.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-15T04:35:43.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:35:43.282 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:35:43.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:35:43.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:35:43.767 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:35:43.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:35:43.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:35:43.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:35:43.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:35:43.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:35:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:43.791+0000 7f0c03329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:43.796 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:43.793+0000 7f0c03329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:43.797 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:43.794+0000 7f0c03329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:35:44.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:35:44.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:35:44.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:35:44.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:35:44.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:35:44.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:44.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:35:44.119 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:35:44.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:44.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:35:44.310 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:44.309+0000 7f0c03329440 -1 Falling back to public interface 2024-09-15T04:35:44.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:45.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:45.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:45.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:35:45.501 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:35:45.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:45.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:35:45.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:46.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:35:46.810+0000 7f0c03329440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:35:46.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:46.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:46.858 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:35:46.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:35:46.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:46.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:35:47.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:35:48.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:35:48.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:35:48.397 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:35:48.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:35:48.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:35:48.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:35:48.746 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/218142,v1:127.0.0.1:6811/218142] [v2:127.0.0.1:6812/218142,v1:127.0.0.1:6813/218142] exists,up 4496238b-0c9d-447b-9e73-d3e113bd2666 2024-09-15T04:35:48.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:35:48.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:35:48.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:35:48.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-09-15T04:35:48.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:35:48.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-09-15T04:35:48.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-15T04:35:49.266 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' already exists 2024-09-15T04:35:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:35:50.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-15T04:35:51.065 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T04:35:51.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:35:51.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-09-15T04:35:51.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-15T04:35:51.739 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test2' already exists 2024-09-15T04:35:51.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:35:52.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-15T04:35:53.479 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 size to 2 2024-09-15T04:35:53.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:35:53.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-09-15T04:35:53.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-15T04:35:54.063 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test3' already exists 2024-09-15T04:35:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:35:55.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-09-15T04:35:55.932 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 3 size to 2 2024-09-15T04:35:55.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-09-15T04:36:00.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-09-15T04:36:00.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:36:00.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:36:00.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:36:00.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:36:00.947 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:36:00.947 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:36:00.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:36:00.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:36:00.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:36:01.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:36:01.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:36:01.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:36:01.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:36:01.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:36:01.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:36:01.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:36:01.397 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T04:36:01.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:36:01.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:36:01.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:36:01.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-15T04:36:01.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-15T04:36:01.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T04:36:01.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:36:01.507 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:36:01.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672965 2024-09-15T04:36:01.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672965 2024-09-15T04:36:01.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487 1-42949672965' 2024-09-15T04:36:01.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:36:01.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-15T04:36:01.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:36:01.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:36:01.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-15T04:36:01.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:36:01.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-15T04:36:01.618 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836487 2024-09-15T04:36:01.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T04:36:01.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:36:01.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-15T04:36:01.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:36:02.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:36:02.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:36:03.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-15T04:36:03.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:36:04.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T04:36:04.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:36:04.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-15T04:36:04.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:36:04.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672965 2024-09-15T04:36:04.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:36:04.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:36:04.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672965 2024-09-15T04:36:04.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:36:04.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672965 2024-09-15T04:36:04.688 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672965 2024-09-15T04:36:04.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-15T04:36:04.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:36:05.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-15T04:36:05.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:36:05.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:36:05.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:36:05.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-15T04:36:05.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:36:05.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:36:05.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:36:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:36:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:36:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:36:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:36:05.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-15T04:36:05.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:36:05.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:36:05.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:36:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-15T04:36:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:36:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:36:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:35:56.101715+0000 0'0 26:17 [1,0] 1 [1,0] 1 0'0 2024-09-15T04:35:53.829744+0000 0'0 2024-09-15T04:35:53.829744+0000 0 0 periodic scrub scheduled @ 2024-09-16T04:43:34.555931+0000 0 0 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:35:53.965194+0000 0'0 25:22 [0,1] 0 [0,1] 0 0'0 2024-09-15T04:35:51.491147+0000 0'0 2024-09-15T04:35:51.491147+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:32:32.447908+0000 0 0 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:35:51.248925+0000 0'0 25:23 [1,0] 1 [1,0] 1 0'0 2024-09-15T04:35:49.047234+0000 0'0 2024-09-15T04:35:49.047234+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:28:30.232354+0000 0 0 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:36:06.730 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:36:06.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-09-15T04:36:06.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-09-15T04:36:06.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-09-15T04:36:06.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-09-15T04:36:06.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:36:06.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-09-15T04:36:06.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-15T04:36:07.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:36:07.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-09-15T04:36:07.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:36:07.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-09-15T04:36:07.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-09-15T04:36:07.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:36:07.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-09-15T04:36:07.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:36:07.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-09-15T04:36:07.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-09-15T04:36:07.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-09-15T04:36:07.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-09-15T04:36:07.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-09-15T04:36:07.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-09-15T04:36:07.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-09-15T04:36:07.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-09-15T04:36:07.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-09-15T04:36:07.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-09-15T04:36:07.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-09-15T04:36:07.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-09-15T04:36:07.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:36:07.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-09-15T04:36:07.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-15T04:36:07.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-15T04:36:08.046 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test3' does not exist 2024-09-15T04:36:08.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-09-15T04:36:08.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-09-15T04:36:08.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-09-15T04:36:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-09-15T04:36:08.058 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-09-15T04:36:08.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-09-15T04:36:08.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-15T04:36:08.894 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 1 2024-09-15T04:36:08.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-09-15T04:36:09.783 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 recovery_priority to 1 2024-09-15T04:36:09.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-09-15T04:36:10.530 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 size to 1 2024-09-15T04:36:10.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-09-15T04:36:11.468 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 recovery_priority to 2 2024-09-15T04:36:11.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-09-15T04:36:11.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:36:11.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:36:11.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:36:11.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:36:11.484 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:36:11.484 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:36:11.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:36:11.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:36:11.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:36:11.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:36:11.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:36:11.949 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T04:36:11.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:36:11.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:36:11.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:36:12.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836492 2024-09-15T04:36:12.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836492 2024-09-15T04:36:12.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492' 2024-09-15T04:36:12.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:36:12.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:36:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672969 2024-09-15T04:36:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672969 2024-09-15T04:36:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836492 1-42949672969' 2024-09-15T04:36:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:36:12.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836492 2024-09-15T04:36:12.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:36:12.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:36:12.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836492 2024-09-15T04:36:12.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:36:12.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836492 2024-09-15T04:36:12.168 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836492 2024-09-15T04:36:12.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-15T04:36:12.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:36:12.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836489 -lt 21474836492 2024-09-15T04:36:12.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:36:13.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:36:13.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:36:13.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-15T04:36:13.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:36:13.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672969 2024-09-15T04:36:13.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:36:13.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:36:13.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672969 2024-09-15T04:36:13.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:36:13.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672969 2024-09-15T04:36:13.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-09-15T04:36:13.876 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672969 2024-09-15T04:36:13.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:36:14.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672970 -lt 42949672969 2024-09-15T04:36:14.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:36:14.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:36:14.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:36:14.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T04:36:14.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:36:14.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:36:14.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:36:14.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:36:14.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:36:14.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:36:14.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:36:15.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:36:15.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:36:15.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:36:15.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:36:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T04:36:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:36:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:36:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-15T04:36:15.621 INFO:tasks.workunit.client.0.smithi138.stderr:10+0 records in 2024-09-15T04:36:15.621 INFO:tasks.workunit.client.0.smithi138.stderr:10+0 records out 2024-09-15T04:36:15.621 INFO:tasks.workunit.client.0.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0291448 s, 360 MB/s 2024-09-15T04:36:15.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-09-15T04:36:15.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-15T04:36:15.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-15T04:36:15.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:15.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-15T04:36:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-15T04:36:16.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:16.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-15T04:36:16.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:16.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-15T04:36:17.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:17.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-15T04:36:17.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:17.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-15T04:36:17.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:17.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-15T04:36:18.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:18.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-15T04:36:18.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:18.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-15T04:36:19.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:19.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-15T04:36:19.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:19.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-15T04:36:19.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:19.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-15T04:36:20.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:20.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-15T04:36:20.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:20.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-15T04:36:20.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:20.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-15T04:36:21.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:21.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-15T04:36:21.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:21.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-15T04:36:22.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:22.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-15T04:36:22.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:22.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-15T04:36:22.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:22.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-15T04:36:23.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:23.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-15T04:36:23.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:23.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-15T04:36:23.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:23.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-15T04:36:24.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:24.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-15T04:36:24.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:24.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-15T04:36:24.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:24.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-15T04:36:25.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:25.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-15T04:36:25.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:25.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-15T04:36:26.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:26.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-15T04:36:26.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:26.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-15T04:36:26.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:26.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-15T04:36:27.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:27.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-15T04:36:27.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:27.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-15T04:36:27.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:27.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-15T04:36:28.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:28.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-15T04:36:28.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:28.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-15T04:36:28.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:28.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-15T04:36:29.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:29.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-15T04:36:29.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:29.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-15T04:36:30.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:30.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-15T04:36:30.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:30.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-15T04:36:30.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:30.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-15T04:36:31.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:31.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-15T04:36:31.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:31.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-15T04:36:32.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:32.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-15T04:36:32.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:32.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-15T04:36:32.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:32.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-15T04:36:33.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:33.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-15T04:36:33.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:33.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-15T04:36:33.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:33.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-15T04:36:34.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:34.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-15T04:36:34.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:34.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-15T04:36:34.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:34.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-15T04:36:35.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:35.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-15T04:36:35.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:35.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-15T04:36:36.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:36.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-15T04:36:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-15T04:36:36.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:36.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-15T04:36:37.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:37.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-15T04:36:37.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:37.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-15T04:36:37.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:37.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-15T04:36:38.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:38.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-15T04:36:38.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:38.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-15T04:36:38.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:38.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-15T04:36:39.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:39.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-15T04:36:39.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:39.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-15T04:36:40.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:40.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-15T04:36:40.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:40.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-15T04:36:40.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:40.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-15T04:36:41.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:41.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-15T04:36:41.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:41.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-15T04:36:42.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:42.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-15T04:36:42.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:42.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-15T04:36:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-15T04:36:43.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:43.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-15T04:36:43.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:43.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-15T04:36:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-15T04:36:44.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-15T04:36:44.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:44.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-15T04:36:45.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:45.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-15T04:36:45.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:45.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-15T04:36:45.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:45.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-15T04:36:46.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:46.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-15T04:36:46.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:46.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-15T04:36:47.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:47.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-15T04:36:47.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:47.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-15T04:36:47.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:47.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-15T04:36:48.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:48.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-15T04:36:48.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:48.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-15T04:36:49.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:49.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-15T04:36:49.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:49.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-15T04:36:49.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:49.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-15T04:36:50.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:50.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-15T04:36:50.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:50.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-15T04:36:50.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:50.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-15T04:36:51.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:51.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-15T04:36:51.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:51.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-15T04:36:52.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:52.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-15T04:36:52.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:52.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-15T04:36:52.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:52.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-15T04:36:53.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:53.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-15T04:36:53.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:53.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-15T04:36:53.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:53.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-15T04:36:54.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:54.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-15T04:36:54.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:54.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-15T04:36:55.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:55.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-15T04:36:55.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:55.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-15T04:36:55.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:55.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-15T04:36:56.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:56.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-15T04:36:56.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:56.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-15T04:36:57.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:57.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-15T04:36:57.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:57.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-15T04:36:57.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:57.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-15T04:36:58.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:58.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-15T04:36:58.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:58.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-15T04:36:58.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:58.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-15T04:36:59.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:59.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-15T04:36:59.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:36:59.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-15T04:37:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-15T04:37:00.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:00.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-15T04:37:00.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:00.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-15T04:37:01.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:01.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-15T04:37:01.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:01.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-15T04:37:02.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:02.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-15T04:37:02.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:02.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-15T04:37:02.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:02.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-15T04:37:03.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:03.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-15T04:37:03.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:03.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-15T04:37:04.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:04.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-15T04:37:04.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:04.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-15T04:37:04.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:04.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-15T04:37:05.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:05.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-15T04:37:05.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:05.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-15T04:37:05.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:05.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-15T04:37:06.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:06.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-15T04:37:06.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:06.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-15T04:37:07.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:07.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-15T04:37:07.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:07.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-15T04:37:07.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:07.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-15T04:37:08.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:08.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-15T04:37:08.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:08.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-15T04:37:08.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:08.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-15T04:37:09.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:09.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-15T04:37:09.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:09.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-15T04:37:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:10.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-15T04:37:10.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:10.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-15T04:37:10.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:10.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-15T04:37:11.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:11.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-15T04:37:11.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:11.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-15T04:37:11.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:11.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-15T04:37:12.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:12.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-15T04:37:12.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:12.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-15T04:37:13.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:13.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-15T04:37:13.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:13.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-15T04:37:13.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:13.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-15T04:37:14.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:14.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-15T04:37:14.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:14.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-15T04:37:14.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:14.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-15T04:37:15.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:15.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-15T04:37:15.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:15.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-15T04:37:16.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:16.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-15T04:37:16.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:16.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-15T04:37:16.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:16.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-15T04:37:17.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:17.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-15T04:37:17.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:17.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-15T04:37:18.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:18.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-15T04:37:18.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:18.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-15T04:37:18.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:18.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-15T04:37:19.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:19.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-15T04:37:19.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:19.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-15T04:37:19.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:19.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-15T04:37:20.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:20.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-15T04:37:20.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:20.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-15T04:37:20.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:20.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-15T04:37:21.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:21.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-15T04:37:21.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:21.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-15T04:37:21.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:21.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-15T04:37:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-15T04:37:22.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:22.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-15T04:37:22.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:22.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-15T04:37:23.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:23.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-15T04:37:23.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:23.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-15T04:37:24.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:24.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-15T04:37:24.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:24.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-15T04:37:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:24.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-15T04:37:25.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:25.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-15T04:37:25.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:25.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-15T04:37:25.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:25.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-15T04:37:26.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:26.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-15T04:37:26.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:26.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-15T04:37:27.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:27.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-15T04:37:27.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:27.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-15T04:37:27.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:27.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-15T04:37:28.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:28.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-15T04:37:28.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:28.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-15T04:37:29.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:29.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-15T04:37:29.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:29.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-15T04:37:29.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:29.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-15T04:37:30.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:30.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-15T04:37:30.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:30.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-15T04:37:30.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-09-15T04:37:30.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-09-15T04:37:30.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-09-15T04:37:30.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-09-15T04:37:30.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:30.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-09-15T04:37:31.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:31.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-09-15T04:37:31.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:31.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-09-15T04:37:32.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:32.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-09-15T04:37:32.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:32.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-09-15T04:37:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:32.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-09-15T04:37:33.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:33.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-09-15T04:37:33.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:33.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-09-15T04:37:34.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:34.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-09-15T04:37:34.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:34.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-09-15T04:37:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-09-15T04:37:35.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:35.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-09-15T04:37:35.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:35.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-09-15T04:37:35.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:35.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-09-15T04:37:36.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:36.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-09-15T04:37:36.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:36.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-09-15T04:37:36.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:36.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-09-15T04:37:37.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:37.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-09-15T04:37:37.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:37.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-09-15T04:37:38.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:38.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-09-15T04:37:38.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:38.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-09-15T04:37:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-09-15T04:37:39.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:39.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-09-15T04:37:39.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:39.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-09-15T04:37:39.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:39.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-09-15T04:37:40.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:40.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-09-15T04:37:40.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:40.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-09-15T04:37:40.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:40.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-09-15T04:37:41.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:41.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-09-15T04:37:41.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:41.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-09-15T04:37:42.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:42.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-09-15T04:37:42.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:42.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-09-15T04:37:42.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:42.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-09-15T04:37:43.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:43.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-09-15T04:37:43.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:43.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-09-15T04:37:43.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:43.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-09-15T04:37:44.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:44.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-09-15T04:37:44.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:44.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-09-15T04:37:44.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:44.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-09-15T04:37:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-09-15T04:37:45.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:45.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-09-15T04:37:46.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:46.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-09-15T04:37:46.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:46.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-09-15T04:37:46.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:46.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-09-15T04:37:47.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:47.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-09-15T04:37:47.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:47.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-09-15T04:37:47.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:47.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-09-15T04:37:48.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:48.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-09-15T04:37:48.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:48.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-09-15T04:37:49.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:49.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-09-15T04:37:49.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:49.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-09-15T04:37:49.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:49.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-09-15T04:37:50.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:50.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-09-15T04:37:50.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:50.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-09-15T04:37:50.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:50.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-09-15T04:37:51.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:51.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-09-15T04:37:51.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:51.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-09-15T04:37:52.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-09-15T04:37:52.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:52.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-09-15T04:37:52.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:52.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-09-15T04:37:53.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:53.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-09-15T04:37:53.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:53.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-09-15T04:37:53.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:53.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-09-15T04:37:54.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:54.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-09-15T04:37:54.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:54.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-09-15T04:37:55.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:55.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-09-15T04:37:55.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:55.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-09-15T04:37:55.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:55.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-09-15T04:37:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-09-15T04:37:56.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:56.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-09-15T04:37:56.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:56.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-09-15T04:37:57.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:57.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-09-15T04:37:57.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:57.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-09-15T04:37:57.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:57.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-09-15T04:37:58.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:58.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-09-15T04:37:58.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:58.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-09-15T04:37:59.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:59.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-09-15T04:37:59.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:59.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-09-15T04:37:59.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:37:59.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-09-15T04:38:00.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:00.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-09-15T04:38:00.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:00.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-09-15T04:38:00.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:00.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-09-15T04:38:01.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:01.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-09-15T04:38:01.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:01.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-09-15T04:38:02.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:02.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-09-15T04:38:02.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:02.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-09-15T04:38:02.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:02.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-09-15T04:38:03.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:03.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-09-15T04:38:03.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:03.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-09-15T04:38:04.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:04.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-09-15T04:38:04.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:04.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-09-15T04:38:04.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:04.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-09-15T04:38:05.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:05.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-09-15T04:38:05.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:05.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-09-15T04:38:05.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:05.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-09-15T04:38:06.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:06.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-09-15T04:38:06.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:06.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-09-15T04:38:07.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:07.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-09-15T04:38:07.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:07.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-09-15T04:38:07.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:07.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-09-15T04:38:08.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:08.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-09-15T04:38:08.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:08.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-09-15T04:38:09.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:09.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-09-15T04:38:09.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:09.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-09-15T04:38:09.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:09.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-09-15T04:38:10.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:10.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-09-15T04:38:10.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:10.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-09-15T04:38:10.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:10.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-09-15T04:38:11.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:11.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-09-15T04:38:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:11.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-09-15T04:38:12.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:12.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-09-15T04:38:12.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:12.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-09-15T04:38:12.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:12.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-09-15T04:38:13.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:13.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-09-15T04:38:13.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:13.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-09-15T04:38:13.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:13.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-09-15T04:38:14.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:14.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-09-15T04:38:14.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:14.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-09-15T04:38:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-09-15T04:38:15.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:15.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-09-15T04:38:15.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:15.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-09-15T04:38:16.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:16.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-09-15T04:38:16.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:16.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-09-15T04:38:17.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:17.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-09-15T04:38:17.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:17.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-09-15T04:38:17.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:17.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-09-15T04:38:18.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:18.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-09-15T04:38:18.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:18.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-09-15T04:38:18.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:18.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-09-15T04:38:19.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:19.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-09-15T04:38:19.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:19.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-09-15T04:38:20.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:20.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-09-15T04:38:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-09-15T04:38:20.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:20.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-09-15T04:38:21.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:21.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-09-15T04:38:21.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:21.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-09-15T04:38:21.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:21.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-09-15T04:38:22.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:22.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-09-15T04:38:22.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:22.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-09-15T04:38:22.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:22.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-09-15T04:38:23.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:23.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-09-15T04:38:23.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:23.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-09-15T04:38:24.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:24.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-09-15T04:38:24.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:24.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-09-15T04:38:24.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:24.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-09-15T04:38:25.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:25.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-09-15T04:38:25.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:25.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-09-15T04:38:25.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:25.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-09-15T04:38:26.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:26.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-09-15T04:38:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:26.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-09-15T04:38:27.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:27.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-09-15T04:38:27.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:27.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-09-15T04:38:27.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:27.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-09-15T04:38:28.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:28.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-09-15T04:38:28.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:28.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-09-15T04:38:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-09-15T04:38:29.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:29.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-09-15T04:38:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-09-15T04:38:30.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:30.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-09-15T04:38:30.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:30.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-09-15T04:38:30.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:30.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-09-15T04:38:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-09-15T04:38:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-09-15T04:38:31.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:31.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-09-15T04:38:32.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:32.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-09-15T04:38:32.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:32.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-09-15T04:38:32.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:32.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-09-15T04:38:33.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:33.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-09-15T04:38:33.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:33.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-09-15T04:38:33.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:33.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-09-15T04:38:34.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:34.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-09-15T04:38:34.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:34.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-09-15T04:38:35.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:35.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-09-15T04:38:35.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:35.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-09-15T04:38:35.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:35.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-09-15T04:38:36.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:36.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-09-15T04:38:36.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:36.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-09-15T04:38:37.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:37.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-09-15T04:38:37.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:37.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-09-15T04:38:37.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:37.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-09-15T04:38:38.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:38.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-09-15T04:38:38.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:38.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-09-15T04:38:38.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:38.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-09-15T04:38:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-09-15T04:38:39.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:39.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-09-15T04:38:40.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:40.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-09-15T04:38:40.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:40.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-09-15T04:38:40.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:40.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-09-15T04:38:41.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:41.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-09-15T04:38:41.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:41.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-09-15T04:38:41.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:41.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-09-15T04:38:42.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:42.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-09-15T04:38:42.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:42.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-09-15T04:38:42.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:42.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-09-15T04:38:43.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:43.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-09-15T04:38:43.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:43.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-09-15T04:38:44.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:44.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-09-15T04:38:44.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:44.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-09-15T04:38:44.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:44.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-09-15T04:38:45.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-09-15T04:38:45.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-09-15T04:38:45.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-09-15T04:38:45.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-15T04:38:45.542 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-15T04:38:45.543 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T04:38:45.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T04:38:45.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-15T04:38:45.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T04:38:46.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-09-15T04:38:46.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-15T04:38:46.652 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:38:46.652 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 189 0 0 0 0 1981808640 0 0 567 0 567 active+clean 2024-09-15T04:36:10.956507+0000 36'567 36:605 [0] 0 [0] 0 0'0 2024-09-15T04:35:51.491147+0000 0'0 2024-09-15T04:35:51.491147+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:22:08.241886+0000 0 0 2024-09-15T04:38:46.652 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:36:09.094958+0000 36'600 36:640 [1] 1 [1] 1 0'0 2024-09-15T04:35:49.047234+0000 0'0 2024-09-15T04:35:49.047234+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:37:41.236111+0000 0 0 2024-09-15T04:38:46.653 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:38:46.653 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:38:46.653 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:46.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-09-15T04:38:46.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-09-15T04:38:47.295 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T04:38:47.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-09-15T04:38:48.083 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 size to 2 2024-09-15T04:38:48.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:36:10.956507+0000 36'600 36:638 [0] 0 [0] 0 0'0 2024-09-15T04:35:51.491147+0000 0'0 2024-09-15T04:35:51.491147+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:22:08.241886+0000 0 0 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 activating+undersized+remapped 2024-09-15T04:38:48.158936+0000 36'600 40:649 [1,0] 1 [1] 1 0'0 2024-09-15T04:35:49.047234+0000 0'0 2024-09-15T04:35:49.047234+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:03:53.233766+0000 0 0 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:38:48.426 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:48.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-09-15T04:38:48.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-09-15T04:38:48.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-15T04:38:48.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-15T04:38:48.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-15T04:38:48.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-15T04:38:48.766 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:48.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-09-15T04:38:48.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-09-15T04:38:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-09-15T04:38:50.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-09-15T04:38:50.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-09-15T04:38:50.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-15T04:38:50.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-15T04:38:50.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-15T04:38:50.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-15T04:38:51.127 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:51.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-09-15T04:38:51.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-09-15T04:38:53.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-09-15T04:38:53.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-09-15T04:38:53.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-09-15T04:38:53.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-09-15T04:38:53.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-09-15T04:38:53.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-09-15T04:38:53.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-09-15T04:38:53.491 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:53.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-09-15T04:38:53.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-09-15T04:38:53.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-09-15T04:38:53.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-09-15T04:38:53.838 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:38:53.838 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-15T04:38:49.823581+0000 36'600 42:655 [0,1] 0 [0] 0 0'0 2024-09-15T04:35:51.491147+0000 0'0 2024-09-15T04:35:51.491147+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:39:32.542833+0000 0 0 2024-09-15T04:38:53.838 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 191 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-09-15T04:38:48.554009+0000 36'600 42:674 [1,0] 1 [1] 1 0'0 2024-09-15T04:35:49.047234+0000 0'0 2024-09-15T04:35:49.047234+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:03:53.233766+0000 0 0 2024-09-15T04:38:53.838 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:38:53.838 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:38:53.839 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:38:53.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-09-15T04:38:53.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:38:53.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-15T04:38:53.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:38:53.930 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 2024-09-15T04:38:53.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-09-15T04:38:53.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-15T04:38:53.931 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:38:53.932 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 191, 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:38:53.933 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "2.0", 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 192, 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:53.934 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:38:53.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-09-15T04:38:53.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T04:38:53.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T04:38:53.936 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:38:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:38:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:38:53.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.0.asok 2024-09-15T04:38:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-09-15T04:38:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.0.asok dump_recovery_reservations 2024-09-15T04:38:54.012 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:38:54.013 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "2.0", 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 192, 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:38:54.014 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 191, 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-15T04:38:54.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-15T04:38:54.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-15T04:38:54.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-09-15T04:38:54.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-15T04:38:54.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-15T04:38:54.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-15T04:38:54.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-09-15T04:38:54.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-15T04:38:54.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-15T04:38:54.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-15T04:38:54.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-09-15T04:38:54.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-09-15T04:38:54.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-09-15T04:38:54.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-15T04:38:54.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-15T04:38:54.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-09-15T04:38:54.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-09-15T04:38:54.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-15T04:38:54.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-09-15T04:38:54.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-15T04:38:54.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-09-15T04:38:54.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-15T04:38:54.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-09-15T04:38:54.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-15T04:38:54.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-09-15T04:38:54.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-15T04:38:54.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-15T04:38:54.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-09-15T04:38:54.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-09-15T04:38:54.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-09-15T04:38:54.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-09-15T04:38:54.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-09-15T04:38:54.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-09-15T04:38:54.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:38:54.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:38:54.041 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:38:54.041 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:38:54.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:38:54.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:38:54.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:38:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:38:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:38:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:38:54.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:38:54.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:38:54.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:38:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:38:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T04:38:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:38:54.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:38:54.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:38:55.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836526 2024-09-15T04:38:55.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836526 2024-09-15T04:38:55.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526' 2024-09-15T04:38:55.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:38:55.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673004 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673004 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836526 1-42949673004' 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836526 2024-09-15T04:38:55.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:38:55.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:38:55.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836526 2024-09-15T04:38:55.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:38:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836526 2024-09-15T04:38:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-09-15T04:38:55.425 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836526 2024-09-15T04:38:55.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:38:56.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836526 -lt 21474836526 2024-09-15T04:38:56.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:38:56.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673004 2024-09-15T04:38:56.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:38:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:38:56.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673004 2024-09-15T04:38:56.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:38:56.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673004 2024-09-15T04:38:56.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673004' 2024-09-15T04:38:56.078 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673004 2024-09-15T04:38:56.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:38:56.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673004 -lt 42949673004 2024-09-15T04:38:56.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:38:56.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:38:56.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:38:57.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T04:38:57.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:38:57.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:38:57.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:38:57.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:38:57.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:38:57.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:38:57.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:38:57.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:38:57.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:38:57.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:38:57.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:38:58.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:38:58.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T04:38:58.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:38:58.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T04:38:58.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:38:58.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:38:58.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:38:58.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:38:58.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:38:58.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:38:58.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:38:58.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:38:59.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:38:59.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:00.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22611898 2024-09-15T04:39:00.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22611898 '!=' null 2024-09-15T04:39:00.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:00.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:00.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:00.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:00.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:00.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:00.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:00.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:00.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:00.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:00.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:00.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:00.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:00.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:00.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:01.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:01.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20094026 2024-09-15T04:39:01.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20094026 '!=' null 2024-09-15T04:39:01.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:01.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:01.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:02.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:02.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:02.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:02.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:02.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:02.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:02.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:02.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:02.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:02.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:02.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:02.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:03.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:03.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32325517 2024-09-15T04:39:03.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32325517 '!=' null 2024-09-15T04:39:03.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:03.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:03.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:03.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:03.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:03.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:03.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:03.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:03.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:03.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:03.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:04.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:04.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:04.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:04.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:04.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:04.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:04.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:04.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:04.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32325517 2024-09-15T04:39:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32325517 '!=' null 2024-09-15T04:39:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:05.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:05.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:05.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:05.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:05.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:05.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:05.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:05.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:06.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:06.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:06.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:06.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:06.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:06.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:07.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588701 2024-09-15T04:39:07.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588701 '!=' null 2024-09-15T04:39:07.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:07.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:07.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:07.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:07.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:07.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:08.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:08.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:08.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:08.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:08.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:08.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:08.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:08.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:09.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841472 2024-09-15T04:39:09.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841472 '!=' null 2024-09-15T04:39:09.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:09.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:09.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:09.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:09.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:09.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:09.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:09.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:09.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:09.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:09.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:10.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:10.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:10.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:10.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:10.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:10.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:10.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:10.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:11.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25335906 2024-09-15T04:39:11.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25335906 '!=' null 2024-09-15T04:39:11.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:11.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:11.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:11.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:11.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:11.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:11.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:11.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:11.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:11.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:12.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:12.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:12.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715334 2024-09-15T04:39:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715334 '!=' null 2024-09-15T04:39:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:13.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:13.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:13.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:13.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:13.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:13.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:13.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:13.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:13.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:13.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:13.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:13.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:14.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:15.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34073180 2024-09-15T04:39:15.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34073180 '!=' null 2024-09-15T04:39:15.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:15.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:15.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:15.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:15.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:15.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:15.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:15.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:15.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:15.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:15.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:15.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:15.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:15.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:15.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:16.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:16.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:16.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841372 2024-09-15T04:39:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841372 '!=' null 2024-09-15T04:39:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:16.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:17.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:17.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:17.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:17.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:17.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:17.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:17.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:17.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:17.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:17.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:17.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:17.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:17.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:17.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:17.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:17.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:18.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083220 2024-09-15T04:39:18.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083220 '!=' null 2024-09-15T04:39:18.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:18.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:18.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:18.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:18.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:18.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:18.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:18.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:19.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:20.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083187 2024-09-15T04:39:20.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083187 '!=' null 2024-09-15T04:39:20.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:20.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:20.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:20.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:20.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:20.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:20.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:20.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:20.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:20.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:20.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:20.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:20.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:21.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083187 2024-09-15T04:39:21.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083187 '!=' null 2024-09-15T04:39:21.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:21.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:21.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:21.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:21.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:21.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:21.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:21.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:21.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:21.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:21.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:22.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:22.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:22.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:22.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:22.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:23.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841511 2024-09-15T04:39:23.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841511 '!=' null 2024-09-15T04:39:23.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:23.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:23.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:23.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:23.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:23.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:24.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:24.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:24.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:24.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:24.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:24.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:24.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:25.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33199055 2024-09-15T04:39:25.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33199055 '!=' null 2024-09-15T04:39:25.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:25.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:25.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:25.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:25.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:25.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:25.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:25.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:25.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:25.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:26.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:26.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:26.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:26.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:26.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:26.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:26.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:26.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:26.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:26.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841494 2024-09-15T04:39:26.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841494 '!=' null 2024-09-15T04:39:26.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:26.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:26.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:27.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:27.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:27.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:27.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:27.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:27.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:27.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:28.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:28.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:28.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083977 2024-09-15T04:39:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083977 '!=' null 2024-09-15T04:39:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:28.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:28.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:28.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:29.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:29.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:29.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:29.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:30.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:30.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:30.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957931 2024-09-15T04:39:30.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957931 '!=' null 2024-09-15T04:39:30.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:30.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:30.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:30.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:30.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:30.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:31.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:31.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:31.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:31.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:31.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:31.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:31.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:32.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715905 2024-09-15T04:39:32.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715905 '!=' null 2024-09-15T04:39:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:32.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:32.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:32.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:32.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:32.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:33.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:33.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34074003 2024-09-15T04:39:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34074003 '!=' null 2024-09-15T04:39:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:33.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:33.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:33.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:33.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:34.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:34.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:34.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:34.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:34.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:34.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:34.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:34.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:35.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34074003 2024-09-15T04:39:35.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34074003 '!=' null 2024-09-15T04:39:35.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:35.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:35.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:35.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:35.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:35.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:35.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:35.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:35.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:35.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:35.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:35.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:35.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:35.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:35.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:36.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:36.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:36.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:36.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:37.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22716016 2024-09-15T04:39:37.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22716016 '!=' null 2024-09-15T04:39:37.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:37.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:37.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:38.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:38.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:38.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:38.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:38.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958168 2024-09-15T04:39:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958168 '!=' null 2024-09-15T04:39:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:39.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:39.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:39.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:39.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:39.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:39.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:39.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:39.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:39.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:39.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:39.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:40.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:40.435 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958163 2024-09-15T04:39:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958163 '!=' null 2024-09-15T04:39:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:41.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:41.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:41.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:41.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:41.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:41.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:41.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:41.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:41.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:41.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:41.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:41.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:41.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:42.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:42.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21842280 2024-09-15T04:39:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21842280 '!=' null 2024-09-15T04:39:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:42.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:42.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:42.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:42.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:42.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:42.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:42.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:42.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:42.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:43.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:43.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:43.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:43.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:43.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:43.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:43.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:43.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:44.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33200204 2024-09-15T04:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33200204 '!=' null 2024-09-15T04:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:44.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:44.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:44.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:44.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:44.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:44.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:44.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:44.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:44.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:44.839 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:44.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:44.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:45.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:45.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:46.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33200204 2024-09-15T04:39:46.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33200204 '!=' null 2024-09-15T04:39:46.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:46.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:46.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:46.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:46.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:46.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:46.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:46.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:46.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:47.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:47.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:47.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:47.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21842207 2024-09-15T04:39:47.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21842207 '!=' null 2024-09-15T04:39:47.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:47.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:47.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:47.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:47.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:47.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:48.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:48.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:48.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:48.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:48.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:48.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:48.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:48.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:49.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957947 2024-09-15T04:39:49.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957947 '!=' null 2024-09-15T04:39:49.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:49.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:49.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:49.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:49.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:49.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:49.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:49.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:50.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:50.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957822 2024-09-15T04:39:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957822 '!=' null 2024-09-15T04:39:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:50.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:50.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:50.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:51.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:51.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:51.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:51.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:51.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:51.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:51.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:51.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:52.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715706 2024-09-15T04:39:52.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715706 '!=' null 2024-09-15T04:39:52.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:52.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:52.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:52.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:52.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:52.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:53.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:53.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:53.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:53.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:53.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:53.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:53.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:54.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34073468 2024-09-15T04:39:54.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34073468 '!=' null 2024-09-15T04:39:54.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:54.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:54.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:54.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:54.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:55.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:55.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:55.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:55.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:55.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:55.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:56.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715598 2024-09-15T04:39:56.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715598 '!=' null 2024-09-15T04:39:56.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:56.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:56.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:56.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:56.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:56.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:56.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:56.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:56.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:56.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:57.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:57.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:57.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:57.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715598 2024-09-15T04:39:57.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715598 '!=' null 2024-09-15T04:39:57.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:57.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:57.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:57.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:57.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:57.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:39:58.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:39:58.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:39:58.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:39:58.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:39:58.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:39:59.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957690 2024-09-15T04:39:59.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957690 '!=' null 2024-09-15T04:39:59.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:39:59.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:39:59.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:39:59.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:39:59.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:39:59.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:00.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:00.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:00.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:00.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:00.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:00.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:00.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957690 2024-09-15T04:40:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957690 '!=' null 2024-09-15T04:40:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:01.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:01.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:01.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:01.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:02.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:02.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:02.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:02.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:02.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:03.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715667 2024-09-15T04:40:03.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715667 '!=' null 2024-09-15T04:40:03.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:03.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:03.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:03.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:03.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:03.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:03.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:03.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:03.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:03.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:03.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:03.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:03.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:03.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:03.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:04.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:04.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:05.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33199999 2024-09-15T04:40:05.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33199999 '!=' null 2024-09-15T04:40:05.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:05.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:05.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:05.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:05.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:05.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:05.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:05.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:05.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:05.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:05.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:05.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:05.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:05.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:05.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:06.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:07.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21842152 2024-09-15T04:40:07.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21842152 '!=' null 2024-09-15T04:40:07.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:07.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:07.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:07.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:07.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:07.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:07.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:07.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:07.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:07.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:07.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:07.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:07.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:07.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:07.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:08.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:08.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:08.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:08.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:08.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:08.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:08.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:08.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:08.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:08.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27084359 2024-09-15T04:40:08.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27084359 '!=' null 2024-09-15T04:40:08.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:08.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:08.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:09.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:09.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:09.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:09.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:09.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:09.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:09.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:10.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:10.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:10.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:10.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210705 2024-09-15T04:40:10.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210705 '!=' null 2024-09-15T04:40:10.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:10.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:10.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:10.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:10.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:10.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:11.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:11.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:11.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:11.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:11.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:11.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:11.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:11.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:11.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:11.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:11.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:11.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:12.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20968581 2024-09-15T04:40:12.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20968581 '!=' null 2024-09-15T04:40:12.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:12.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:12.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:12.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:12.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:12.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:12.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:12.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:12.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:12.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:13.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:13.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:13.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:14.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32326444 2024-09-15T04:40:14.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32326444 '!=' null 2024-09-15T04:40:14.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:14.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:14.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:14.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:14.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:14.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:14.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:14.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:14.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:14.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:14.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:14.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:14.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:14.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:15.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:15.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:15.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:15.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32326444 2024-09-15T04:40:15.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32326444 '!=' null 2024-09-15T04:40:15.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:15.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:15.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:15.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:15.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:15.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:15.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:15.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:15.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:15.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:15.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:16.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:16.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:16.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:16.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:16.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:16.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:17.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21842184 2024-09-15T04:40:17.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21842184 '!=' null 2024-09-15T04:40:17.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:17.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:17.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:17.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:17.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:17.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:17.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:17.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:17.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:17.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:17.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:18.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:18.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:18.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:18.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:18.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:18.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:18.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:18.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958090 2024-09-15T04:40:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958090 '!=' null 2024-09-15T04:40:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:19.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:19.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:19.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:19.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:19.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:19.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:19.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:20.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:20.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27958069 2024-09-15T04:40:20.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27958069 '!=' null 2024-09-15T04:40:20.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:20.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:20.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:20.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:20.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:20.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:21.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:21.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:21.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:21.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:21.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:21.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715599 2024-09-15T04:40:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715599 '!=' null 2024-09-15T04:40:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:22.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:22.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:22.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:22.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:22.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:22.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:22.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:22.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:23.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:23.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:23.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:23.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:23.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:23.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:23.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:23.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:23.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:23.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:23.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:23.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:23.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=34073234 2024-09-15T04:40:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 34073234 '!=' null 2024-09-15T04:40:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:24.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:24.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:24.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:24.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:24.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:24.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:24.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:24.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:24.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:25.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:25.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:25.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:25.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:25.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:25.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:25.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:25.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:25.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:25.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715562 2024-09-15T04:40:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715562 '!=' null 2024-09-15T04:40:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:26.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:26.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:26.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:26.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:26.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:26.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:26.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:26.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:26.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:26.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:26.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:26.887 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:27.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:27.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:27.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:27.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:27.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:27.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:27.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:27.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:27.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:28.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28831244 2024-09-15T04:40:28.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28831244 '!=' null 2024-09-15T04:40:28.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:28.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:28.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:28.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:28.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:28.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:28.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:28.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:28.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:28.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:28.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:28.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:29.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:29.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957503 2024-09-15T04:40:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957503 '!=' null 2024-09-15T04:40:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:30.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:30.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:30.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:30.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:30.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:30.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:30.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:31.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:31.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:31.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:31.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957503 2024-09-15T04:40:31.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957503 '!=' null 2024-09-15T04:40:31.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:31.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:31.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:31.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:31.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:32.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:32.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:32.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:32.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:32.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:33.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715204 2024-09-15T04:40:33.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715204 '!=' null 2024-09-15T04:40:33.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:33.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:33.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:33.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:33.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:33.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:33.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:33.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:33.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:33.864 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:34.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:34.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:34.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33199476 2024-09-15T04:40:34.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33199476 '!=' null 2024-09-15T04:40:34.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:34.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:34.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:35.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:35.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:35.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:35.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:35.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:35.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:35.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:35.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:35.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:36.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841815 2024-09-15T04:40:36.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841815 '!=' null 2024-09-15T04:40:36.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:36.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:36.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:36.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:36.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:36.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:37.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:37.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:37.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:37.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083893 2024-09-15T04:40:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083893 '!=' null 2024-09-15T04:40:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:38.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:38.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:38.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:38.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:38.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:38.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:38.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:38.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:38.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:38.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:38.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:38.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:39.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:39.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:39.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:40.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210041 2024-09-15T04:40:40.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210041 '!=' null 2024-09-15T04:40:40.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:40.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:40.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:40.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:40.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:40.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:40.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:40.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:40.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:40.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:41.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:42.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20967968 2024-09-15T04:40:42.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20967968 '!=' null 2024-09-15T04:40:42.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:42.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:42.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:42.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:42.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:42.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:42.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:42.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:43.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20967968 2024-09-15T04:40:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20967968 '!=' null 2024-09-15T04:40:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:43.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:43.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:44.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:44.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:44.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:44.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:44.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:44.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:44.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:44.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:44.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:44.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:44.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:44.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:44.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32325703 2024-09-15T04:40:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32325703 '!=' null 2024-09-15T04:40:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:45.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:45.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:45.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:45.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:45.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:45.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:45.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:45.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:46.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841769 2024-09-15T04:40:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841769 '!=' null 2024-09-15T04:40:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:47.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:47.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:47.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:47.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:47.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:48.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957500 2024-09-15T04:40:48.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957500 '!=' null 2024-09-15T04:40:48.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:48.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:48.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:48.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:48.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:48.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:49.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:49.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:49.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:49.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:49.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:49.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:49.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:49.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:50.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957376 2024-09-15T04:40:50.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957376 '!=' null 2024-09-15T04:40:50.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:50.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:50.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:50.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:50.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:50.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:50.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:50.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:51.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:51.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:51.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:51.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:51.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:51.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:51.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:51.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:51.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:52.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22714956 2024-09-15T04:40:52.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22714956 '!=' null 2024-09-15T04:40:52.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:52.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:52.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:52.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:52.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:52.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:52.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:52.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:53.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:53.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:53.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:53.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:53.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33198860 2024-09-15T04:40:53.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33198860 '!=' null 2024-09-15T04:40:53.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:53.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:53.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:53.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:53.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:53.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:53.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:53.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:53.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:53.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:53.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:54.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:54.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:54.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:54.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:54.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:55.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33198860 2024-09-15T04:40:55.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33198860 '!=' null 2024-09-15T04:40:55.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:55.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:55.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:55.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:55.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:55.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:56.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:56.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:56.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:57.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841538 2024-09-15T04:40:57.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841538 '!=' null 2024-09-15T04:40:57.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:57.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:57.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:57.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:57.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:57.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:57.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:57.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:57.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:57.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:57.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:58.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:58.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:58.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:58.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:40:58.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:40:58.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:40:58.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:40:58.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:40:58.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:40:58.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:40:58.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:40:58.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:40:58.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:40:59.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957042 2024-09-15T04:40:59.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957042 '!=' null 2024-09-15T04:40:59.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:40:59.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:40:59.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:40:59.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:40:59.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:40:59.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:40:59.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:40:59.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:40:59.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:40:59.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:00.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:00.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:00.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:00.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:00.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27956625 2024-09-15T04:41:00.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27956625 '!=' null 2024-09-15T04:41:00.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:00.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:00.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:01.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:01.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:01.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:01.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:01.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:01.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:01.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:01.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:01.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:02.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22714722 2024-09-15T04:41:02.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22714722 '!=' null 2024-09-15T04:41:02.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:02.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:02.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:02.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:02.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:02.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:02.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:02.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:02.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:02.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:02.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:03.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:03.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:03.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:03.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:03.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:03.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:03.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:04.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33199243 2024-09-15T04:41:04.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33199243 '!=' null 2024-09-15T04:41:04.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:04.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:04.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:04.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:04.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:04.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:04.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:04.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:04.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:04.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:04.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:04.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:04.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:04.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:04.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:05.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:05.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:05.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:05.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33199243 2024-09-15T04:41:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33199243 '!=' null 2024-09-15T04:41:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:06.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:06.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:06.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:06.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:06.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:07.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:07.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:08.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957275 2024-09-15T04:41:08.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957275 '!=' null 2024-09-15T04:41:08.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:08.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:08.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:08.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:08.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:08.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:08.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:08.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:08.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:08.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:08.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:08.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:08.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:08.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:08.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:08.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:08.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:08.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:08.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:09.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957275 2024-09-15T04:41:09.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957275 '!=' null 2024-09-15T04:41:09.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:09.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:09.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:09.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:09.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:09.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:10.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:10.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:10.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:10.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:10.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:11.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083725 2024-09-15T04:41:11.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083725 '!=' null 2024-09-15T04:41:11.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:11.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:11.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:11.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:11.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:11.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:11.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:11.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:11.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:11.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:12.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:12.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:13.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20968320 2024-09-15T04:41:13.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20968320 '!=' null 2024-09-15T04:41:13.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:13.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:13.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:13.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:13.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:13.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:13.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:13.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:13.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:13.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:13.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:13.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:13.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:13.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:13.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:14.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:14.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:15.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=32326174 2024-09-15T04:41:15.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 32326174 '!=' null 2024-09-15T04:41:15.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:15.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:15.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:15.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:15.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:15.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:15.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:16.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:16.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:16.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:16.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21841976 2024-09-15T04:41:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21841976 '!=' null 2024-09-15T04:41:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:16.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:16.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:16.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:16.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:16.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:16.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:16.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:16.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:16.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:17.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:17.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:17.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:17.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:17.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:17.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:17.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:18.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957866 2024-09-15T04:41:18.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957866 '!=' null 2024-09-15T04:41:18.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:18.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:18.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:18.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:18.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:18.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:18.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:18.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:19.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:19.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:19.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:20.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27957648 2024-09-15T04:41:20.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27957648 '!=' null 2024-09-15T04:41:20.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:20.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:20.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:20.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:20.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:20.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:20.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:20.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:20.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:20.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:20.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:20.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:20.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:20.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:21.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:21.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:21.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:22.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22715327 2024-09-15T04:41:22.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22715327 '!=' null 2024-09-15T04:41:22.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:22.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:22.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:22.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:22.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:22.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:22.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:22.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:22.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:22.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:22.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:22.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:22.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:22.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:22.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:23.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:23.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:23.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:23.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:23.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:23.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210050 2024-09-15T04:41:23.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210050 '!=' null 2024-09-15T04:41:23.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:23.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:23.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:24.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:24.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:24.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:24.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:24.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:24.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:24.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:24.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:24.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:24.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:24.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:25.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210050 2024-09-15T04:41:25.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210050 '!=' null 2024-09-15T04:41:25.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:25.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:25.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:25.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:25.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:25.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:25.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:25.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:25.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:26.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:26.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:26.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:27.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588767 2024-09-15T04:41:27.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588767 '!=' null 2024-09-15T04:41:27.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:27.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:27.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:27.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:27.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:27.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:27.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:27.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:27.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:27.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:27.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:27.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:27.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:27.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:27.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:27.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:28.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=33642537 2024-09-15T04:41:28.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 33642537 '!=' null 2024-09-15T04:41:28.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:28.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:28.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:28.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:28.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:28.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:41:28.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:28.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:28.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:29.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-09-15T04:41:29.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:29.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:29.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13978666 2024-09-15T04:41:29.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13978666 '!=' null 2024-09-15T04:41:29.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:29.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:29.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:29.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:29.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:29.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:29.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:29.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:29.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:29.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:29.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:30.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:41:30.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:30.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:30.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:30.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-09-15T04:41:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-09-15T04:41:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:41:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-15T04:41:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:30.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:30.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:30.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:30.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:30.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:30.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:30.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:30.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:31.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:41:31.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:31.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:31.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-09-15T04:41:31.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:31.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:32.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11144352 2024-09-15T04:41:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11144352 '!=' null 2024-09-15T04:41:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:32.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:32.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:32.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:32.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:32.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:32.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:32.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:32.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:32.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:41:32.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:32.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:32.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:41:33.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:41:33.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:41:33.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:41:33.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11144352 2024-09-15T04:41:33.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11144352 '!=' null 2024-09-15T04:41:33.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:41:33.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T04:41:33.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:41:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:41:33.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:41:33.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:41:33.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:41:33.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:41:33.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:41:33.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:41:34.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T04:41:34.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:41:34.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:41:34.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-09-15T04:41:34.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-09-15T04:41:34.518 INFO:tasks.workunit.client.0.smithi138.stdout:TEST PASSED 2024-09-15T04:41:34.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-09-15T04:41:34.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-15T04:41:34.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-15T04:41:34.985 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' does not exist 2024-09-15T04:41:34.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-09-15T04:41:34.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-15T04:41:34.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-15T04:41:35.657 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test2' does not exist 2024-09-15T04:41:35.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-09-15T04:41:35.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:41:35.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:41:35.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:41:35.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:41:35.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:41:36.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:41:36.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:41:36.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:41:36.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:41:36.984 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:41:36.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:41:36.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:41:36.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:41:36.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:41:36.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:41:36.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:41:36.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:41:36.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:41:36.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:41:36.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-15T04:41:37.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:41:37.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215650 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-09-15T04:41:37.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:41:37.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:41:37.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:41:37.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:41:37.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:41:37.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:41:37.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:41:37.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:41:37.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:41:37.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:41:37.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:41:37.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:41:37.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:41:37.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:41:37.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:41:37.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-15T04:41:37.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:41:37.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215650 2024-09-15T04:41:37.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:41:37.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:41:37.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-09-15T04:41:37.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:41:37.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.215650 2024-09-15T04:41:37.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:41:37.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:41:37.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-09-15T04:41:37.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:37.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:41:37.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:41:37.660 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:41:37.661 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:41:37.661 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:41:37.661 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.661 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.662 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-mon.a.asok 2024-09-15T04:41:37.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:41:37.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215650/ceph-mon.a.asok config get fsid 2024-09-15T04:41:37.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:37.740 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:37.741 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-mon.a.asok 2024-09-15T04:41:37.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:41:37.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.215650/ceph-mon.a.asok config get mon_host 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-09-15T04:41:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:41:38.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:41:38.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:41:38.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:41:38.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:41:38.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-09-15T04:41:38.680 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-09-15T04:41:38.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-09-15T04:41:38.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:41:38.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-09-15T04:41:38.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:41:38.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:41:38.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:41:38.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:41:38.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:41:38.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8f931e4e-ee5b-4b89-9f07-a36f178c3065 2024-09-15T04:41:38.689 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 8f931e4e-ee5b-4b89-9f07-a36f178c3065 2024-09-15T04:41:38.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8f931e4e-ee5b-4b89-9f07-a36f178c3065' 2024-09-15T04:41:38.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:41:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCZeZmtm3NKRAAuwZuAlhgMSy2jlI7OkoHCQ== 2024-09-15T04:41:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCZeZmtm3NKRAAuwZuAlhgMSy2jlI7OkoHCQ=="}' 2024-09-15T04:41:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8f931e4e-ee5b-4b89-9f07-a36f178c3065 -i td/osd-recovery-prio/0/new.json 2024-09-15T04:41:38.999 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:41:39.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-09-15T04:41:39.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCZeZmtm3NKRAAuwZuAlhgMSy2jlI7OkoHCQ== --osd-uuid 8f931e4e-ee5b-4b89-9f07-a36f178c3065 2024-09-15T04:41:39.036 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.036+0000 7fada1875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:39.045 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.044+0000 7fada1875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.045+0000 7fada1875440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.046+0000 7fada1875440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:41:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.046+0000 7fada1875440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:41:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.046+0000 7fada1875440 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-09-15T04:41:39.085 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:39.084+0000 7fada1875440 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-09-15T04:41:44.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-09-15T04:41:44.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:41:44.428 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:41:44.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:41:44.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:41:44.908 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:41:44.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:41:44.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:41:44.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:41:44.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:41:44.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:41:44.933 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:44.932+0000 7f7f387fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:44.934 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:44.934+0000 7f7f387fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:44.935 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:44.935+0000 7f7f387fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:45.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:41:45.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:41:45.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:41:45.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:41:45.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:41:45.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:45.270 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:41:45.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:41:45.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:45.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:41:45.453 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:45.452+0000 7f7f387fe440 -1 Falling back to public interface 2024-09-15T04:41:45.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:41:46.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:41:46.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:46.622 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:41:46.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:41:46.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:46.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:41:46.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:41:47.925 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:47.924+0000 7f7f387fe440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:41:47.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:41:47.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:47.981 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:41:47.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:41:47.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:47.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:41:48.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:41:49.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:41:49.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:49.412 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:41:49.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:41:49.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:49.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:41:49.771 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/236411,v1:127.0.0.1:6803/236411] [v2:127.0.0.1:6804/236411,v1:127.0.0.1:6805/236411] exists,up 8f931e4e-ee5b-4b89-9f07-a36f178c3065 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:41:49.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:41:49.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:41:49.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98cc90ed-e13a-4944-8f79-1802f9598d63 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 98cc90ed-e13a-4944-8f79-1802f9598d63 2024-09-15T04:41:49.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 98cc90ed-e13a-4944-8f79-1802f9598d63' 2024-09-15T04:41:49.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:41:49.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCNZeZmAWoYLxAADXRhSQWCLLJAnLsy4X2IGg== 2024-09-15T04:41:49.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCNZeZmAWoYLxAADXRhSQWCLLJAnLsy4X2IGg=="}' 2024-09-15T04:41:49.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98cc90ed-e13a-4944-8f79-1802f9598d63 -i td/osd-recovery-prio/1/new.json 2024-09-15T04:41:50.314 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:41:50.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-09-15T04:41:50.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNZeZmAWoYLxAADXRhSQWCLLJAnLsy4X2IGg== --osd-uuid 98cc90ed-e13a-4944-8f79-1802f9598d63 2024-09-15T04:41:50.354 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.353+0000 7f39e2a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:50.374 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.373+0000 7f39e2a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:50.375 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.374+0000 7f39e2a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:50.375 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.375+0000 7f39e2a75440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:41:50.376 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.375+0000 7f39e2a75440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:41:50.376 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.375+0000 7f39e2a75440 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-09-15T04:41:50.466 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:50.465+0000 7f39e2a75440 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-09-15T04:41:56.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-09-15T04:41:56.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:41:56.153 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:41:56.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:41:56.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:41:56.654 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:41:56.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:41:56.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:41:56.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:41:56.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:41:56.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:41:56.679 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:56.678+0000 7fa9b2871440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:56.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:56.680+0000 7fa9b2871440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:56.683 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:56.681+0000 7fa9b2871440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:57.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:41:57.049 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:41:57.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:57.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:41:57.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:57.201+0000 7fa9b2871440 -1 Falling back to public interface 2024-09-15T04:41:57.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:41:58.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:41:58.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:58.404 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:41:58.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:41:58.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:58.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:41:58.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:41:59.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:41:59.677+0000 7fa9b2871440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:41:59.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:41:59.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:41:59.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:41:59.759 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:41:59.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:41:59.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:42:00.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:01.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:01.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:01.283 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:42:01.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:42:01.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:01.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:42:01.641 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/237810,v1:127.0.0.1:6811/237810] [v2:127.0.0.1:6812/237810,v1:127.0.0.1:6813/237810] exists,up 98cc90ed-e13a-4944-8f79-1802f9598d63 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:42:01.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:42:01.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:42:01.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=451d4631-14d8-4cd6-a911-616c90a2b56b 2024-09-15T04:42:01.648 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 451d4631-14d8-4cd6-a911-616c90a2b56b 2024-09-15T04:42:01.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 451d4631-14d8-4cd6-a911-616c90a2b56b' 2024-09-15T04:42:01.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:42:01.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCZZeZmocRcJxAA7n5EWLmveNwGoE7SbdT3jg== 2024-09-15T04:42:01.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCZZeZmocRcJxAA7n5EWLmveNwGoE7SbdT3jg=="}' 2024-09-15T04:42:01.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 451d4631-14d8-4cd6-a911-616c90a2b56b -i td/osd-recovery-prio/2/new.json 2024-09-15T04:42:02.128 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:42:02.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-09-15T04:42:02.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCZZeZmocRcJxAA7n5EWLmveNwGoE7SbdT3jg== --osd-uuid 451d4631-14d8-4cd6-a911-616c90a2b56b 2024-09-15T04:42:02.166 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.165+0000 7fceb4e2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:02.167 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.167+0000 7fceb4e2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:02.169 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.168+0000 7fceb4e2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:02.169 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.168+0000 7fceb4e2c440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-15T04:42:02.169 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.168+0000 7fceb4e2c440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-15T04:42:02.169 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.168+0000 7fceb4e2c440 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-09-15T04:42:02.261 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:02.260+0000 7fceb4e2c440 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-09-15T04:42:08.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-09-15T04:42:08.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:42:08.171 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:42:08.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:42:08.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:42:08.659 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:42:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:42:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:42:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:42:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:42:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:42:08.684 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:08.684+0000 7f8773a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:08.686 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:08.686+0000 7f8773a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:08.687 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:08.687+0000 7f8773a2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:09.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:42:09.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:42:09.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:42:09.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:42:09.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:42:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:42:09.054 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:42:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:09.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:42:09.201 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:09.201+0000 7f8773a2d440 -1 Falling back to public interface 2024-09-15T04:42:09.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:10.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:10.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:10.402 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:42:10.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:42:10.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:10.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:42:10.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:11.678 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:11.677+0000 7f8773a2d440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:42:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:11.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:42:11.762 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:42:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:11.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:42:12.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:13.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:13.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:13.293 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:42:13.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:42:13.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:13.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:42:13.652 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/239209,v1:127.0.0.1:6819/239209] [v2:127.0.0.1:6820/239209,v1:127.0.0.1:6821/239209] exists,up 451d4631-14d8-4cd6-a911-616c90a2b56b 2024-09-15T04:42:13.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:42:13.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:42:13.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:42:13.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:42:13.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c 2024-09-15T04:42:13.660 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c 2024-09-15T04:42:13.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c' 2024-09-15T04:42:13.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:42:13.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQClZeZmwBERKBAAaynpjgg+RhTeISKWaiO0vg== 2024-09-15T04:42:13.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQClZeZmwBERKBAAaynpjgg+RhTeISKWaiO0vg=="}' 2024-09-15T04:42:13.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c -i td/osd-recovery-prio/3/new.json 2024-09-15T04:42:14.221 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:42:14.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-09-15T04:42:14.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQClZeZmwBERKBAAaynpjgg+RhTeISKWaiO0vg== --osd-uuid 4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c 2024-09-15T04:42:14.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.258+0000 7fb13ea0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:14.260 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.260+0000 7fb13ea0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:14.261 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.261+0000 7fb13ea0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:14.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.261+0000 7fb13ea0e440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-15T04:42:14.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.261+0000 7fb13ea0e440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-15T04:42:14.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.261+0000 7fb13ea0e440 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-09-15T04:42:14.362 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:14.361+0000 7fb13ea0e440 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-09-15T04:42:20.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-09-15T04:42:20.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:42:20.257 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T04:42:20.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T04:42:20.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:42:20.742 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:42:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T04:42:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:42:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:42:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:42:20.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:42:20.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:20.767+0000 7f270120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:20.769+0000 7f270120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:20.770 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:20.770+0000 7f270120c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:42:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:21.092 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:42:21.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:42:21.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:21.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:42:21.288 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:21.287+0000 7f270120c440 -1 Falling back to public interface 2024-09-15T04:42:21.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:22.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:22.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:22.445 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:42:22.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:42:22.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:22.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:42:22.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:23.773 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:23.773+0000 7f270120c440 -1 osd.3 0 log_to_monitors true 2024-09-15T04:42:23.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:23.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:23.806 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:42:23.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:42:23.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:23.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:42:24.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:25.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:25.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:25.209 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:42:25.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:42:25.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:25.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:42:25.564 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/240608,v1:127.0.0.1:6827/240608] [v2:127.0.0.1:6828/240608,v1:127.0.0.1:6829/240608] exists,up 4c47af65-f64e-4eb2-bdd6-8dcd0d4ba48c 2024-09-15T04:42:25.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:42:25.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:42:25.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:42:25.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:42:25.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:42:25.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:42:25.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd14a002-3f17-41b7-8f3b-06ad1b10962a 2024-09-15T04:42:25.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 dd14a002-3f17-41b7-8f3b-06ad1b10962a' 2024-09-15T04:42:25.572 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 dd14a002-3f17-41b7-8f3b-06ad1b10962a 2024-09-15T04:42:25.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:42:25.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCxZeZmRE6+IhAAOi2ypnS0Anbtb6iWvdkjyA== 2024-09-15T04:42:25.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCxZeZmRE6+IhAAOi2ypnS0Anbtb6iWvdkjyA=="}' 2024-09-15T04:42:25.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd14a002-3f17-41b7-8f3b-06ad1b10962a -i td/osd-recovery-prio/4/new.json 2024-09-15T04:42:26.049 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:42:26.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-09-15T04:42:26.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCxZeZmRE6+IhAAOi2ypnS0Anbtb6iWvdkjyA== --osd-uuid dd14a002-3f17-41b7-8f3b-06ad1b10962a 2024-09-15T04:42:26.087 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.087+0000 7f42dbe40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:26.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.088+0000 7f42dbe40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.089+0000 7f42dbe40440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.090+0000 7f42dbe40440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-15T04:42:26.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.090+0000 7f42dbe40440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-15T04:42:26.091 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.090+0000 7f42dbe40440 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-09-15T04:42:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:26.139+0000 7f42dbe40440 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-09-15T04:42:32.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-09-15T04:42:32.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:42:32.279 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T04:42:32.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T04:42:32.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:42:32.761 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T04:42:32.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T04:42:32.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=b4750794-ba90-444a-a7f0-0f06a9694b2b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.215650/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:42:32.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:42:32.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:42:32.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:42:32.787 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:32.786+0000 7fcdd2921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:32.789 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:32.788+0000 7fcdd2921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:32.790 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:32.789+0000 7fcdd2921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:42:33.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:33.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:42:33.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:33.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:42:33.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:33.303+0000 7fcdd2921440 -1 Falling back to public interface 2024-09-15T04:42:33.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:34.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:34.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:34.508 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:42:34.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:42:34.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:34.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:42:34.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:35.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:42:35.810+0000 7fcdd2921440 -1 osd.4 0 log_to_monitors true 2024-09-15T04:42:35.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:35.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:35.886 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:42:35.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:42:35.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:35.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:42:36.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:42:37.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:42:37.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:42:37.288 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:42:37.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:42:37.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:42:37.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:42:37.646 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/242007,v1:127.0.0.1:6835/242007] [v2:127.0.0.1:6836/242007,v1:127.0.0.1:6837/242007] exists,up dd14a002-3f17-41b7-8f3b-06ad1b10962a 2024-09-15T04:42:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:42:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:42:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:42:37.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-09-15T04:42:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-09-15T04:42:37.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-15T04:42:38.171 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' already exists 2024-09-15T04:42:38.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:39.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-09-15T04:42:39.822 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T04:42:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-09-15T04:42:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-09-15T04:42:40.408 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test2' already exists 2024-09-15T04:42:40.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:41.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-09-15T04:42:42.391 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 size to 2 2024-09-15T04:42:42.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:42.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-09-15T04:42:42.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-09-15T04:42:42.991 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test3' already exists 2024-09-15T04:42:43.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-15T04:42:45.175 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 3 size to 2 2024-09-15T04:42:45.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:45.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-09-15T04:42:45.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-09-15T04:42:46.116 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test4' already exists 2024-09-15T04:42:46.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:47.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-09-15T04:42:48.200 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 4 size to 2 2024-09-15T04:42:48.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:48.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-09-15T04:42:48.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-09-15T04:42:48.905 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test5' already exists 2024-09-15T04:42:48.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:49.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-09-15T04:42:51.477 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 5 size to 2 2024-09-15T04:42:51.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:51.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-09-15T04:42:51.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-09-15T04:42:51.958 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test6' already exists 2024-09-15T04:42:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:52.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-09-15T04:42:54.053 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 6 size to 2 2024-09-15T04:42:54.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:54.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-09-15T04:42:54.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-09-15T04:42:54.779 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test7' already exists 2024-09-15T04:42:54.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:55.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-15T04:42:56.789 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 7 size to 2 2024-09-15T04:42:56.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:56.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-09-15T04:42:56.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-09-15T04:42:57.462 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test8' already exists 2024-09-15T04:42:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:42:58.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-09-15T04:42:59.474 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 8 size to 2 2024-09-15T04:42:59.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:42:59.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-09-15T04:42:59.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-09-15T04:43:00.217 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test9' already exists 2024-09-15T04:43:00.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:43:01.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-09-15T04:43:02.186 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 9 size to 2 2024-09-15T04:43:02.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:02.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-09-15T04:43:02.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-09-15T04:43:02.847 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test10' already exists 2024-09-15T04:43:02.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:43:03.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-09-15T04:43:04.887 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 10 size to 2 2024-09-15T04:43:04.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-09-15T04:43:09.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-09-15T04:43:09.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:43:09.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:43:09.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:43:09.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:43:09.901 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:43:09.901 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:43:09.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:43:09.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:43:09.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:43:09.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:43:10.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:43:10.370 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:43:10.371 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:43:10.371 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:43:10.371 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:43:10.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:43:10.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:10.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:43:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T04:43:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T04:43:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T04:43:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:10.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:43:10.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672976 2024-09-15T04:43:10.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672976 2024-09-15T04:43:10.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-15T04:43:10.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:10.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:43:10.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509454 2024-09-15T04:43:10.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509454 2024-09-15T04:43:10.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509454' 2024-09-15T04:43:10.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:10.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:43:10.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345932 2024-09-15T04:43:10.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345932 2024-09-15T04:43:10.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509454 3-85899345932' 2024-09-15T04:43:10.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:10.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182409 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182409 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509454 3-85899345932 4-107374182409' 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T04:43:10.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:10.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:43:10.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T04:43:10.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:10.968 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836500 2024-09-15T04:43:10.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T04:43:10.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T04:43:10.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:43:11.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-15T04:43:11.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:43:12.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:43:12.320 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:43:12.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T04:43:12.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:12.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672976 2024-09-15T04:43:12.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:12.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:43:12.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672976 2024-09-15T04:43:12.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:12.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672976 2024-09-15T04:43:12.679 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672976 2024-09-15T04:43:12.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-15T04:43:12.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:43:13.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672977 -lt 42949672976 2024-09-15T04:43:13.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:13.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509454 2024-09-15T04:43:13.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:13.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:43:13.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509454 2024-09-15T04:43:13.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:13.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509454 2024-09-15T04:43:13.036 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509454 2024-09-15T04:43:13.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-09-15T04:43:13.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:43:13.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509454 -lt 64424509454 2024-09-15T04:43:13.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:13.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345932 2024-09-15T04:43:13.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:13.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:43:13.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345932 2024-09-15T04:43:13.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:13.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345932 2024-09-15T04:43:13.391 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345932 2024-09-15T04:43:13.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-09-15T04:43:13.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:43:13.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-15T04:43:13.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:13.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182409 2024-09-15T04:43:13.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:13.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:43:13.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182409 2024-09-15T04:43:13.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:13.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182409 2024-09-15T04:43:13.769 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182409 2024-09-15T04:43:13.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2024-09-15T04:43:13.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:43:14.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182409 -lt 107374182409 2024-09-15T04:43:14.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:43:14.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:43:14.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:43:14.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:43:14.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-09-15T04:43:14.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:43:14.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:43:14.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:43:15.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-09-15T04:43:15.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:43:15.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:43:15.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-09-15T04:43:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:43:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:43:05.541356+0000 0'0 81:15 [2,1] 2 [2,1] 2 0'0 2024-09-15T04:43:02.612899+0000 0'0 2024-09-15T04:43:02.612899+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:29:11.609667+0000 0 0 2024-09-15T04:43:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:43:02.606236+0000 0'0 81:26 [1,2] 1 [1,2] 1 0'0 2024-09-15T04:42:59.970445+0000 0'0 2024-09-15T04:42:59.970445+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:24:56.365303+0000 0 0 2024-09-15T04:43:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:43:00.288279+0000 0'0 78:26 [3,4] 3 [3,4] 3 0'0 2024-09-15T04:42:57.160272+0000 0'0 2024-09-15T04:42:57.160272+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:56:00.866953+0000 0 0 2024-09-15T04:43:15.749 INFO:tasks.workunit.client.0.smithi138.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:57.192688+0000 0'0 81:36 [1,4] 1 [1,4] 1 0'0 2024-09-15T04:42:54.403964+0000 0'0 2024-09-15T04:42:54.403964+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:51:41.258153+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:55.017575+0000 0'0 81:34 [4,2] 4 [4,2] 4 0'0 2024-09-15T04:42:51.797040+0000 0'0 2024-09-15T04:42:51.797040+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:13:44.807427+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:52.133900+0000 0'0 78:38 [3,4] 3 [3,4] 3 0'0 2024-09-15T04:42:48.605322+0000 0'0 2024-09-15T04:42:48.605322+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:00:39.756303+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:49.311287+0000 0'0 81:38 [4,0] 4 [4,0] 4 0'0 2024-09-15T04:42:45.669200+0000 0'0 2024-09-15T04:42:45.669200+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:32:53.510038+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:45.751194+0000 0'0 81:48 [1,2] 1 [1,2] 1 0'0 2024-09-15T04:42:42.819245+0000 0'0 2024-09-15T04:42:42.819245+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:39:35.565272+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:44.687260+0000 0'0 78:50 [3,1] 3 [3,1] 3 0'0 2024-09-15T04:42:40.162371+0000 0'0 2024-09-15T04:42:40.162371+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:02:34.473645+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T04:42:40.941433+0000 0'0 81:53 [1,0] 1 [1,0] 1 0'0 2024-09-15T04:42:37.950625+0000 0'0 2024-09-15T04:42:37.950625+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:33:09.656921+0000 0 0 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:43:15.750 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-09-15T04:43:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-09-15T04:43:15.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-09-15T04:43:15.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:15.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-09-15T04:43:15.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:16.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-15T04:43:16.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-09-15T04:43:16.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:16.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-15T04:43:16.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-09-15T04:43:16.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:16.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-15T04:43:16.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-09-15T04:43:16.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-09-15T04:43:16.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:16.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-09-15T04:43:16.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:17.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-15T04:43:17.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-09-15T04:43:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:17.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-09-15T04:43:17.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-15T04:43:17.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:17.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-09-15T04:43:17.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-15T04:43:17.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:17.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-09-15T04:43:17.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:17.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-09-15T04:43:17.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-09-15T04:43:17.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-09-15T04:43:18.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-09-15T04:43:18.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-09-15T04:43:18.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-09-15T04:43:18.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-09-15T04:43:18.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-09-15T04:43:18.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-09-15T04:43:18.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-09-15T04:43:18.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-09-15T04:43:18.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-09-15T04:43:18.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:18.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-09-15T04:43:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-09-15T04:43:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-09-15T04:43:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-09-15T04:43:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-09-15T04:43:18.814 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test2' does not exist 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-09-15T04:43:18.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-09-15T04:43:18.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-09-15T04:43:19.442 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test4' does not exist 2024-09-15T04:43:19.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:19.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-09-15T04:43:19.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-09-15T04:43:19.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-09-15T04:43:19.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-09-15T04:43:20.017 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test5' does not exist 2024-09-15T04:43:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-09-15T04:43:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-09-15T04:43:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-09-15T04:43:20.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-09-15T04:43:20.507 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test6' does not exist 2024-09-15T04:43:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-09-15T04:43:20.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-09-15T04:43:21.316 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test8' does not exist 2024-09-15T04:43:21.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:21.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-09-15T04:43:21.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-09-15T04:43:21.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-09-15T04:43:21.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-09-15T04:43:21.924 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test9' does not exist 2024-09-15T04:43:21.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-09-15T04:43:21.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-09-15T04:43:21.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-09-15T04:43:21.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-09-15T04:43:21.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-09-15T04:43:22.623 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test10' does not exist 2024-09-15T04:43:22.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-09-15T04:43:23.404 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 3 size to 1 2024-09-15T04:43:23.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-09-15T04:43:24.536 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 7 size to 1 2024-09-15T04:43:24.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-09-15T04:43:24.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:43:24.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:43:24.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:43:24.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:43:24.548 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:43:24.548 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:43:24.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:43:24.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:43:24.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:43:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:43:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:43:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:43:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:43:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:43:24.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:43:25.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:43:25.109 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:43:25.109 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:43:25.110 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:43:25.110 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:43:25.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:43:25.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:25.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:43:25.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836505 2024-09-15T04:43:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836505 2024-09-15T04:43:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505' 2024-09-15T04:43:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:25.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:43:25.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672981 2024-09-15T04:43:25.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672981 2024-09-15T04:43:25.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672981' 2024-09-15T04:43:25.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:25.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:43:25.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509459 2024-09-15T04:43:25.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509459 2024-09-15T04:43:25.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672981 2-64424509459' 2024-09-15T04:43:25.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:25.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:43:25.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345937 2024-09-15T04:43:25.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345937 2024-09-15T04:43:25.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672981 2-64424509459 3-85899345937' 2024-09-15T04:43:25.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:43:25.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:43:25.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182414 2024-09-15T04:43:25.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182414 2024-09-15T04:43:25.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836505 1-42949672981 2-64424509459 3-85899345937 4-107374182414' 2024-09-15T04:43:25.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:25.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836505 2024-09-15T04:43:25.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:25.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:43:25.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836505 2024-09-15T04:43:25.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:25.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836505 2024-09-15T04:43:25.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-15T04:43:25.652 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836505 2024-09-15T04:43:25.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:43:25.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836502 -lt 21474836505 2024-09-15T04:43:25.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:43:27.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:43:27.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:43:27.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-15T04:43:27.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:27.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672981 2024-09-15T04:43:27.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:27.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:43:27.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672981 2024-09-15T04:43:27.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:27.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672981 2024-09-15T04:43:27.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-09-15T04:43:27.360 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672981 2024-09-15T04:43:27.361 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:43:27.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672981 2024-09-15T04:43:27.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:27.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509459 2024-09-15T04:43:27.715 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:27.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:43:27.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509459 2024-09-15T04:43:27.716 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:27.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509459 2024-09-15T04:43:27.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-15T04:43:27.718 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509459 2024-09-15T04:43:27.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:43:28.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509459 -lt 64424509459 2024-09-15T04:43:28.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:28.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345937 2024-09-15T04:43:28.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:28.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:43:28.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345937 2024-09-15T04:43:28.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:28.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345937 2024-09-15T04:43:28.074 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345937 2024-09-15T04:43:28.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345937' 2024-09-15T04:43:28.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:43:28.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345937 -lt 85899345937 2024-09-15T04:43:28.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:43:28.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182414 2024-09-15T04:43:28.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:43:28.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:43:28.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182414 2024-09-15T04:43:28.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:43:28.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182414 2024-09-15T04:43:28.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-09-15T04:43:28.470 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182414 2024-09-15T04:43:28.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:43:28.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182414 2024-09-15T04:43:28.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:43:28.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:43:28.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:43:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-15T04:43:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:43:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:43:29.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-15T04:43:29.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:43:29.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:43:29.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:43:30.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-15T04:43:30.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:43:30.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:43:30.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr:10+0 records in 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr:10+0 records out 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0291724 s, 359 MB/s 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-15T04:43:30.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-15T04:43:30.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:30.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-09-15T04:43:30.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:30.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-09-15T04:43:31.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:31.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-09-15T04:43:32.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:32.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-09-15T04:43:32.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:32.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-09-15T04:43:33.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-09-15T04:43:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-09-15T04:43:34.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:34.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-09-15T04:43:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-09-15T04:43:36.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:36.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-09-15T04:43:36.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:36.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-09-15T04:43:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-09-15T04:43:38.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:38.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-09-15T04:43:38.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:38.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-09-15T04:43:39.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:39.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-09-15T04:43:40.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:40.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-09-15T04:43:40.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:40.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-09-15T04:43:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-09-15T04:43:42.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:42.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-09-15T04:43:42.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:42.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-09-15T04:43:43.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:43.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-09-15T04:43:44.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:44.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-09-15T04:43:44.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:44.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-09-15T04:43:45.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:45.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-09-15T04:43:46.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:46.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-09-15T04:43:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:46.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-09-15T04:43:47.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:47.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-09-15T04:43:48.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:48.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-09-15T04:43:48.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:48.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-09-15T04:43:49.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:49.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-09-15T04:43:50.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:50.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-09-15T04:43:50.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:50.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-09-15T04:43:51.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:51.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-09-15T04:43:52.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:52.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-09-15T04:43:52.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:52.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-09-15T04:43:53.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:53.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-09-15T04:43:54.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:54.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-09-15T04:43:54.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:54.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-09-15T04:43:55.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:55.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-09-15T04:43:56.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:56.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-09-15T04:43:56.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:56.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-09-15T04:43:57.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:57.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-09-15T04:43:58.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:58.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-09-15T04:43:58.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:58.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-09-15T04:43:59.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:43:59.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-09-15T04:44:00.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:00.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-09-15T04:44:00.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:00.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-09-15T04:44:01.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:01.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-09-15T04:44:02.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:02.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-09-15T04:44:02.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:02.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-09-15T04:44:03.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:03.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-09-15T04:44:04.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:04.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-09-15T04:44:04.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:04.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-09-15T04:44:05.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:05.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-09-15T04:44:06.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:06.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-09-15T04:44:06.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:06.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-09-15T04:44:07.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:07.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-09-15T04:44:08.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:08.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-09-15T04:44:08.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:08.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-09-15T04:44:09.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:09.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-09-15T04:44:10.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:10.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-09-15T04:44:10.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:10.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-09-15T04:44:11.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:11.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-09-15T04:44:12.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:12.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-09-15T04:44:12.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:12.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-09-15T04:44:13.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:13.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-09-15T04:44:14.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:14.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-09-15T04:44:14.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:14.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-09-15T04:44:15.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:15.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-09-15T04:44:16.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:16.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-09-15T04:44:16.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:16.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-09-15T04:44:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:17.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-09-15T04:44:18.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:18.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-09-15T04:44:18.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:18.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-09-15T04:44:19.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:19.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-09-15T04:44:20.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:20.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-09-15T04:44:20.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:20.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-09-15T04:44:21.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:21.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-09-15T04:44:22.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:22.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-09-15T04:44:22.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:22.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-09-15T04:44:23.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:23.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-09-15T04:44:24.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:24.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-09-15T04:44:24.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:24.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-09-15T04:44:25.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:25.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-09-15T04:44:26.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:26.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-09-15T04:44:27.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:27.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-09-15T04:44:27.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:27.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-09-15T04:44:28.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:28.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-09-15T04:44:29.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:29.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-09-15T04:44:29.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:29.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-09-15T04:44:30.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:30.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-09-15T04:44:31.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:31.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-09-15T04:44:31.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:31.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-09-15T04:44:32.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:32.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-09-15T04:44:33.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:33.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-09-15T04:44:33.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:33.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-09-15T04:44:34.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:34.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-09-15T04:44:35.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:35.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-09-15T04:44:35.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:35.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-09-15T04:44:36.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:36.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-09-15T04:44:37.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:37.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-09-15T04:44:37.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:37.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-09-15T04:44:38.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:38.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-09-15T04:44:39.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:39.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-09-15T04:44:39.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:39.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-09-15T04:44:40.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:40.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-09-15T04:44:41.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:41.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-09-15T04:44:41.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:41.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-09-15T04:44:42.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:42.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-09-15T04:44:43.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:43.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-09-15T04:44:43.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:43.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-09-15T04:44:44.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:44.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-09-15T04:44:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-09-15T04:44:45.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:45.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-09-15T04:44:46.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:46.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-09-15T04:44:47.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:47.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-09-15T04:44:47.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:47.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-09-15T04:44:48.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:48.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-09-15T04:44:49.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:49.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-09-15T04:44:49.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:49.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-09-15T04:44:50.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:50.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-09-15T04:44:51.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:51.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-09-15T04:44:52.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:52.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-09-15T04:44:52.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:52.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-09-15T04:44:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:53.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-09-15T04:44:54.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:54.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-09-15T04:44:54.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:54.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-09-15T04:44:55.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:55.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-09-15T04:44:56.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:56.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-09-15T04:44:56.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:56.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-09-15T04:44:57.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:57.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-09-15T04:44:58.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:58.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-09-15T04:44:58.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:58.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-09-15T04:44:59.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:44:59.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-09-15T04:45:00.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:00.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-09-15T04:45:00.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:00.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-09-15T04:45:01.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:01.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-09-15T04:45:02.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:02.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-09-15T04:45:02.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:02.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-09-15T04:45:03.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:03.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-09-15T04:45:04.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:04.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-09-15T04:45:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:05.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-09-15T04:45:05.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:05.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-09-15T04:45:06.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:06.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-09-15T04:45:07.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:07.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-09-15T04:45:07.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:07.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-09-15T04:45:08.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:08.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-09-15T04:45:09.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:09.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-09-15T04:45:09.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:09.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-09-15T04:45:10.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:10.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-09-15T04:45:11.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:11.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-09-15T04:45:11.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:11.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-09-15T04:45:12.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:12.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-09-15T04:45:13.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:13.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-09-15T04:45:13.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:13.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-09-15T04:45:14.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:14.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-09-15T04:45:15.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:15.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-09-15T04:45:15.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:15.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-09-15T04:45:16.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:16.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-09-15T04:45:17.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:17.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-09-15T04:45:17.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:17.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-09-15T04:45:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-09-15T04:45:19.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:19.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-09-15T04:45:19.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:19.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-09-15T04:45:20.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:20.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-09-15T04:45:21.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:21.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-09-15T04:45:21.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:21.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-09-15T04:45:22.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:22.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-09-15T04:45:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-09-15T04:45:23.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:23.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-09-15T04:45:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-09-15T04:45:25.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:25.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-09-15T04:45:25.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:25.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-09-15T04:45:26.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:26.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-09-15T04:45:27.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:27.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-09-15T04:45:27.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:27.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-09-15T04:45:28.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:28.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-09-15T04:45:29.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:29.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-09-15T04:45:29.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:29.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-09-15T04:45:30.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:30.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-09-15T04:45:31.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:31.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-09-15T04:45:31.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:31.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-09-15T04:45:32.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:32.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-09-15T04:45:33.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:33.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-09-15T04:45:34.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:34.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-09-15T04:45:34.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:34.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-09-15T04:45:35.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:35.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-09-15T04:45:36.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:36.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-09-15T04:45:36.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:36.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-09-15T04:45:37.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:37.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-09-15T04:45:38.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:38.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-09-15T04:45:38.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:38.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-09-15T04:45:39.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:39.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-09-15T04:45:40.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:40.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-09-15T04:45:40.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:40.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-09-15T04:45:41.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:41.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-09-15T04:45:42.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:42.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-09-15T04:45:42.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:42.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-09-15T04:45:43.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:43.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-09-15T04:45:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-09-15T04:45:44.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-09-15T04:45:44.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-09-15T04:45:44.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-15T04:45:44.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-15T04:45:44.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:44.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-09-15T04:45:45.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:45.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-09-15T04:45:45.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:45.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-09-15T04:45:45.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:45.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-09-15T04:45:46.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:46.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-09-15T04:45:46.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:46.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-09-15T04:45:47.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:47.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-09-15T04:45:47.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:47.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-09-15T04:45:47.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:47.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-09-15T04:45:48.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:48.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-09-15T04:45:48.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:48.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-09-15T04:45:48.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:48.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-09-15T04:45:49.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:49.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-09-15T04:45:49.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:49.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-09-15T04:45:49.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:49.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-09-15T04:45:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-09-15T04:45:50.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:50.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-09-15T04:45:50.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:50.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-09-15T04:45:51.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:51.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-09-15T04:45:51.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:51.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-09-15T04:45:52.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:52.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-09-15T04:45:52.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:52.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-09-15T04:45:52.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:52.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-09-15T04:45:53.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:53.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-09-15T04:45:53.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:53.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-09-15T04:45:54.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:54.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-09-15T04:45:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-09-15T04:45:54.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:54.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-09-15T04:45:55.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:55.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-09-15T04:45:55.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:55.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-09-15T04:45:55.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:55.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-09-15T04:45:56.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:56.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-09-15T04:45:56.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:56.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-09-15T04:45:57.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:57.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-09-15T04:45:57.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:57.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-09-15T04:45:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-09-15T04:45:58.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:58.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-09-15T04:45:58.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:58.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-09-15T04:45:58.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:58.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-09-15T04:45:59.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:59.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-09-15T04:45:59.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:59.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-09-15T04:45:59.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:45:59.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-09-15T04:46:00.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:00.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-09-15T04:46:00.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:00.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-09-15T04:46:01.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:01.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-09-15T04:46:01.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:01.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-09-15T04:46:01.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:01.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-09-15T04:46:02.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:02.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-09-15T04:46:02.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:02.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-09-15T04:46:03.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:03.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-09-15T04:46:03.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:03.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-09-15T04:46:03.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:03.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-09-15T04:46:04.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:04.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-09-15T04:46:04.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:04.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-09-15T04:46:04.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:04.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-09-15T04:46:05.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:05.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-09-15T04:46:05.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:05.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-09-15T04:46:05.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:05.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-09-15T04:46:06.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:06.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-09-15T04:46:06.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:06.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-09-15T04:46:07.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:07.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-09-15T04:46:07.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:07.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-09-15T04:46:07.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:07.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-09-15T04:46:08.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:08.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-09-15T04:46:08.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:08.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-09-15T04:46:08.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:08.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-09-15T04:46:09.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:09.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-09-15T04:46:09.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:09.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-09-15T04:46:09.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:09.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-09-15T04:46:10.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:10.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-09-15T04:46:10.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:10.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-09-15T04:46:11.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:11.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-09-15T04:46:11.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:11.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-09-15T04:46:11.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:11.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-09-15T04:46:12.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:12.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-09-15T04:46:12.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:12.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-09-15T04:46:12.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:12.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-09-15T04:46:13.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:13.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-09-15T04:46:13.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:13.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-09-15T04:46:13.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:13.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-09-15T04:46:14.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:14.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-09-15T04:46:14.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:14.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-09-15T04:46:15.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:15.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-09-15T04:46:15.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:15.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-09-15T04:46:15.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:15.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-09-15T04:46:16.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:16.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-09-15T04:46:16.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:16.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-09-15T04:46:17.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:17.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-09-15T04:46:17.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:17.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-09-15T04:46:17.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:17.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-09-15T04:46:17.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:17.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-09-15T04:46:18.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:18.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-09-15T04:46:18.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:18.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-09-15T04:46:19.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:19.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-09-15T04:46:19.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:19.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-09-15T04:46:19.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:19.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-09-15T04:46:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-09-15T04:46:20.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:20.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-09-15T04:46:21.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:21.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-09-15T04:46:21.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:21.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-09-15T04:46:21.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:21.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-09-15T04:46:22.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:22.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-09-15T04:46:22.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:22.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-09-15T04:46:23.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:23.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-09-15T04:46:23.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:23.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-09-15T04:46:23.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:23.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-09-15T04:46:24.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:24.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-09-15T04:46:24.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:24.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-09-15T04:46:24.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:24.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-09-15T04:46:25.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:25.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-09-15T04:46:25.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:25.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-09-15T04:46:25.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:25.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-09-15T04:46:26.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:26.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-09-15T04:46:26.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:26.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-09-15T04:46:27.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:27.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-09-15T04:46:27.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:27.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-09-15T04:46:27.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:27.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-09-15T04:46:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-09-15T04:46:28.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:28.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-09-15T04:46:29.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:29.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-09-15T04:46:29.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:29.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-09-15T04:46:29.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:29.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-09-15T04:46:30.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:30.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-09-15T04:46:30.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:30.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-09-15T04:46:30.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:30.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-09-15T04:46:31.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:31.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-09-15T04:46:31.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:31.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-09-15T04:46:31.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:31.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-09-15T04:46:32.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:32.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-09-15T04:46:32.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:32.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-09-15T04:46:33.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:33.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-09-15T04:46:33.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:33.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-09-15T04:46:33.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:33.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-09-15T04:46:34.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:34.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-09-15T04:46:34.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:34.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-09-15T04:46:35.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:35.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-09-15T04:46:35.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:35.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-09-15T04:46:35.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:35.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-09-15T04:46:36.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:36.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-09-15T04:46:36.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:36.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-09-15T04:46:36.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:36.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-09-15T04:46:37.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:37.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-09-15T04:46:37.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:37.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-09-15T04:46:37.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:37.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-09-15T04:46:38.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:38.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-09-15T04:46:38.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:38.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-09-15T04:46:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-09-15T04:46:39.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:39.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-09-15T04:46:39.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:39.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-09-15T04:46:40.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:40.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-09-15T04:46:40.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:40.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-09-15T04:46:41.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:41.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-09-15T04:46:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:41.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-09-15T04:46:41.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:41.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-09-15T04:46:42.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:42.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-09-15T04:46:42.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:42.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-09-15T04:46:42.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:42.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-09-15T04:46:43.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:43.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-09-15T04:46:43.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:43.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-09-15T04:46:43.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:43.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-09-15T04:46:44.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:44.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-09-15T04:46:44.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:44.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-09-15T04:46:45.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:45.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-09-15T04:46:45.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:45.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-09-15T04:46:45.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:45.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-09-15T04:46:46.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:46.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-09-15T04:46:46.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:46.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-09-15T04:46:46.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:46.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-09-15T04:46:47.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:47.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-09-15T04:46:47.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:47.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-09-15T04:46:47.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:47.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-09-15T04:46:48.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:48.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-09-15T04:46:48.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:48.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-09-15T04:46:49.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:49.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-09-15T04:46:49.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:49.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-09-15T04:46:49.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:49.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-09-15T04:46:50.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:50.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-09-15T04:46:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:50.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-09-15T04:46:50.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:50.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-09-15T04:46:51.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:51.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-09-15T04:46:51.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:51.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-09-15T04:46:51.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:51.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-09-15T04:46:52.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:52.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-09-15T04:46:52.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:52.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-09-15T04:46:53.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:53.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-09-15T04:46:53.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:53.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-09-15T04:46:53.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:53.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-09-15T04:46:54.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:54.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-09-15T04:46:54.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:54.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-09-15T04:46:55.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:55.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-09-15T04:46:55.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:55.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-09-15T04:46:55.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:55.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-09-15T04:46:56.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:56.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-09-15T04:46:56.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:56.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-09-15T04:46:56.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:56.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-09-15T04:46:57.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:57.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-09-15T04:46:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-09-15T04:46:57.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:57.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-09-15T04:46:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:58.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-09-15T04:46:58.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:58.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-09-15T04:46:59.161 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-09-15T04:46:59.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-09-15T04:46:59.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-09-15T04:46:59.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-09-15T04:46:59.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:59.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-09-15T04:46:59.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:59.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-09-15T04:46:59.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:46:59.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-09-15T04:47:00.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:00.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-09-15T04:47:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:00.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-09-15T04:47:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-09-15T04:47:01.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:01.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-09-15T04:47:01.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:01.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-09-15T04:47:02.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:02.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-09-15T04:47:02.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:02.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-09-15T04:47:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-09-15T04:47:03.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:03.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-09-15T04:47:03.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:03.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-09-15T04:47:03.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:03.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-09-15T04:47:04.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:04.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-09-15T04:47:04.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:04.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-09-15T04:47:05.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:05.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-09-15T04:47:05.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:05.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-09-15T04:47:05.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:05.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-09-15T04:47:06.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:06.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-09-15T04:47:06.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:06.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-09-15T04:47:06.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:06.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-09-15T04:47:07.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:07.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-09-15T04:47:07.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:07.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-09-15T04:47:07.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:07.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-09-15T04:47:08.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:08.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-09-15T04:47:08.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:08.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-09-15T04:47:09.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:09.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-09-15T04:47:09.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:09.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-09-15T04:47:09.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:09.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-09-15T04:47:10.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:10.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-09-15T04:47:10.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:10.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-09-15T04:47:10.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:10.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-09-15T04:47:11.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:11.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-09-15T04:47:11.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:11.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-09-15T04:47:11.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:11.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-09-15T04:47:12.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:12.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-09-15T04:47:12.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:12.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-09-15T04:47:13.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:13.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-09-15T04:47:13.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:13.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-09-15T04:47:13.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:13.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-09-15T04:47:14.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:14.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-09-15T04:47:14.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:14.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-09-15T04:47:14.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:14.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-09-15T04:47:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-09-15T04:47:15.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:15.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-09-15T04:47:15.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:15.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-09-15T04:47:16.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:16.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-09-15T04:47:16.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:16.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-09-15T04:47:17.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:17.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-09-15T04:47:17.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:17.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-09-15T04:47:17.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:17.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-09-15T04:47:18.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:18.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-09-15T04:47:18.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:18.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-09-15T04:47:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-09-15T04:47:19.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:19.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-09-15T04:47:19.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:19.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-09-15T04:47:19.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:19.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-09-15T04:47:20.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:20.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-09-15T04:47:20.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:20.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-09-15T04:47:20.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:20.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-09-15T04:47:21.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:21.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-09-15T04:47:21.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:21.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-09-15T04:47:21.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:21.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-09-15T04:47:22.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:22.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-09-15T04:47:22.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:22.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-09-15T04:47:23.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:23.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-09-15T04:47:23.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:23.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-09-15T04:47:23.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:23.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-09-15T04:47:24.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:24.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-09-15T04:47:24.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:24.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-09-15T04:47:25.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:25.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-09-15T04:47:25.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:25.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-09-15T04:47:25.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:25.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-09-15T04:47:26.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:26.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-09-15T04:47:26.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:26.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-09-15T04:47:26.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:26.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-09-15T04:47:27.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:27.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-09-15T04:47:27.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:27.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-09-15T04:47:27.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:27.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-09-15T04:47:28.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:28.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-09-15T04:47:28.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:28.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-09-15T04:47:29.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:29.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-09-15T04:47:29.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:29.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-09-15T04:47:29.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:29.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-09-15T04:47:30.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:30.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-09-15T04:47:30.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:30.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-09-15T04:47:31.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:31.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-09-15T04:47:31.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:31.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-09-15T04:47:31.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:31.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-09-15T04:47:32.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:32.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-09-15T04:47:32.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:32.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-09-15T04:47:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:33.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-09-15T04:47:33.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:33.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-09-15T04:47:33.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:33.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-09-15T04:47:34.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:34.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-09-15T04:47:34.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:34.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-09-15T04:47:34.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:34.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-09-15T04:47:35.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:35.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-09-15T04:47:35.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:35.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-09-15T04:47:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-09-15T04:47:36.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:36.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-09-15T04:47:36.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:36.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-09-15T04:47:37.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:37.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-09-15T04:47:37.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:37.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-09-15T04:47:37.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:37.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-09-15T04:47:38.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:38.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-09-15T04:47:38.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:38.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-09-15T04:47:38.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:38.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-09-15T04:47:39.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:39.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-09-15T04:47:39.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:39.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-09-15T04:47:39.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:39.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-09-15T04:47:40.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:40.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-09-15T04:47:40.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:40.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-09-15T04:47:41.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:41.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-09-15T04:47:41.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:41.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-09-15T04:47:41.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:41.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-09-15T04:47:42.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:42.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-09-15T04:47:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-09-15T04:47:43.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:43.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-09-15T04:47:43.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:43.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-09-15T04:47:43.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:43.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-09-15T04:47:44.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:44.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-09-15T04:47:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-09-15T04:47:44.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:44.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-09-15T04:47:45.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:45.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-09-15T04:47:45.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:45.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-09-15T04:47:45.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:45.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-09-15T04:47:46.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:46.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-09-15T04:47:46.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:46.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-09-15T04:47:47.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:47.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-09-15T04:47:47.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:47.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-09-15T04:47:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:47.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-09-15T04:47:48.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:48.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-09-15T04:47:48.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:48.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-09-15T04:47:48.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:48.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-09-15T04:47:49.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:49.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-09-15T04:47:49.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:49.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-09-15T04:47:49.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:49.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-09-15T04:47:50.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:50.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-09-15T04:47:50.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:50.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-09-15T04:47:51.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:51.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-09-15T04:47:51.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:51.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-09-15T04:47:51.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:51.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-09-15T04:47:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-09-15T04:47:52.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:52.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-09-15T04:47:52.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:52.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-09-15T04:47:53.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:53.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-09-15T04:47:53.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:53.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-09-15T04:47:54.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:54.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-09-15T04:47:54.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:54.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-09-15T04:47:54.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:54.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-09-15T04:47:55.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:55.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-09-15T04:47:55.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:55.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-09-15T04:47:56.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:56.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-09-15T04:47:56.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:56.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-09-15T04:47:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-09-15T04:47:57.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:57.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-09-15T04:47:57.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:57.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-09-15T04:47:57.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:57.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-09-15T04:47:58.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:58.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-09-15T04:47:58.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:58.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-09-15T04:47:59.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:59.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-09-15T04:47:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-09-15T04:47:59.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:47:59.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-09-15T04:48:00.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:00.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-09-15T04:48:00.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:00.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-09-15T04:48:01.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:01.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-09-15T04:48:01.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:01.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-09-15T04:48:01.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:01.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-09-15T04:48:02.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:02.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-09-15T04:48:02.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:02.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-09-15T04:48:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-09-15T04:48:03.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:03.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-09-15T04:48:03.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:03.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-09-15T04:48:04.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:04.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-09-15T04:48:04.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:04.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-09-15T04:48:04.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:04.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-09-15T04:48:05.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:05.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-09-15T04:48:05.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:05.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-09-15T04:48:05.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:05.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-09-15T04:48:06.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:06.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-09-15T04:48:06.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:06.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-09-15T04:48:07.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:07.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-09-15T04:48:07.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:07.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-09-15T04:48:07.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:07.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-09-15T04:48:08.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:08.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-09-15T04:48:08.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:08.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-09-15T04:48:09.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:09.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-09-15T04:48:09.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:09.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-09-15T04:48:09.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:09.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-09-15T04:48:10.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:10.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-09-15T04:48:10.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:10.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-09-15T04:48:10.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:10.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-09-15T04:48:11.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:11.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-09-15T04:48:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:11.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-09-15T04:48:12.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:12.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-09-15T04:48:12.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:12.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-09-15T04:48:12.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:12.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-09-15T04:48:13.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-09-15T04:48:13.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-09-15T04:48:13.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-09-15T04:48:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-09-15T04:48:13.555 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-09-15T04:48:13.555 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T04:48:13.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T04:48:13.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-09-15T04:48:13.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T04:48:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-09-15T04:48:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-09-15T04:48:14.599 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stdout:7.0 193 0 0 0 0 2023751680 0 0 579 0 579 active+clean 2024-09-15T04:43:24.871929+0000 95'579 95:634 [1] 1 [1] 1 0'0 2024-09-15T04:42:54.403964+0000 0'0 2024-09-15T04:42:54.403964+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:24:50.136707+0000 0 0 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:43:23.991206+0000 95'600 95:667 [1] 1 [1] 1 0'0 2024-09-15T04:42:42.819245+0000 0'0 2024-09-15T04:42:42.819245+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:42:02.515009+0000 0 0 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:42:40.941433+0000 95'600 95:664 [1,0] 1 [1,0] 1 0'0 2024-09-15T04:42:37.950625+0000 0'0 2024-09-15T04:42:37.950625+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:33:09.656921+0000 0 0 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:48:14.600 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:48:14.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-09-15T04:48:14.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-09-15T04:48:15.406 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T04:48:15.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-09-15T04:48:16.194 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T04:48:16.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-09-15T04:48:16.975 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 7 size to 2 2024-09-15T04:48:16.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:18.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:18.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:18.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:19.070 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:19.071 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:19.072 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:19.078 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-09-15T04:48:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-15T04:48:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-15T04:48:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-15T04:48:19.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-09-15T04:48:19.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-09-15T04:48:21.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-15T04:48:21.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-15T04:48:21.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-15T04:48:21.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 2 = 10 ']' 2024-09-15T04:48:21.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-09-15T04:48:23.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-15T04:48:23.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-15T04:48:23.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-15T04:48:24.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-09-15T04:48:24.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-09-15T04:48:24.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:48:24.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:48:24.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:48:24.495 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:24.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:48:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836567 2024-09-15T04:48:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836567 2024-09-15T04:48:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567' 2024-09-15T04:48:24.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:24.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:48:24.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673043 2024-09-15T04:48:24.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673043 2024-09-15T04:48:24.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673043' 2024-09-15T04:48:24.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:24.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:48:24.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509521 2024-09-15T04:48:24.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509521 2024-09-15T04:48:24.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509521' 2024-09-15T04:48:24.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:24.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:48:24.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345998 2024-09-15T04:48:24.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345998 2024-09-15T04:48:24.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509521 3-85899345998' 2024-09-15T04:48:24.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:24.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:48:25.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182476 2024-09-15T04:48:25.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182476 2024-09-15T04:48:25.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836567 1-42949673043 2-64424509521 3-85899345998 4-107374182476' 2024-09-15T04:48:25.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:25.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836567 2024-09-15T04:48:25.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:25.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:48:25.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836567 2024-09-15T04:48:25.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:25.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836567 2024-09-15T04:48:25.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836567' 2024-09-15T04:48:25.076 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836567 2024-09-15T04:48:25.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:25.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836564 -lt 21474836567 2024-09-15T04:48:25.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:48:26.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:48:26.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:26.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836567 -lt 21474836567 2024-09-15T04:48:26.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:26.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673043 2024-09-15T04:48:26.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:48:26.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673043 2024-09-15T04:48:26.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:26.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673043 2024-09-15T04:48:26.793 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673043 2024-09-15T04:48:26.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673043' 2024-09-15T04:48:26.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:48:27.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673044 -lt 42949673043 2024-09-15T04:48:27.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:27.180 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509521 2024-09-15T04:48:27.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:27.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:48:27.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509521 2024-09-15T04:48:27.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:27.183 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509521 2024-09-15T04:48:27.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509521 2024-09-15T04:48:27.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509521' 2024-09-15T04:48:27.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:48:27.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509521 -lt 64424509521 2024-09-15T04:48:27.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:27.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345998 2024-09-15T04:48:27.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:27.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:48:27.538 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345998 2024-09-15T04:48:27.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:27.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345998 2024-09-15T04:48:27.539 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345998 2024-09-15T04:48:27.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345998' 2024-09-15T04:48:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:48:27.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345999 -lt 85899345998 2024-09-15T04:48:27.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:27.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182476 2024-09-15T04:48:27.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:27.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:48:27.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182476 2024-09-15T04:48:27.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:27.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182476 2024-09-15T04:48:27.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182476' 2024-09-15T04:48:27.903 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182476 2024-09-15T04:48:27.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:48:28.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182476 -lt 107374182476 2024-09-15T04:48:28.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:28.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:28.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:28.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:28.326 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:28.327 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:28.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-09-15T04:48:28.829 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T04:48:28.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-09-15T04:48:30.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-09-15T04:48:30.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:48:30.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:48:31.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:48:31.203 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:48:31.203 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:48:31.204 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:48:31.204 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:48:31.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:48:31.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:31.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:48:31.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836570 2024-09-15T04:48:31.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836570 2024-09-15T04:48:31.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836570' 2024-09-15T04:48:31.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:31.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:48:31.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673047 2024-09-15T04:48:31.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673047 2024-09-15T04:48:31.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836570 1-42949673047' 2024-09-15T04:48:31.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:31.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:48:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509524 2024-09-15T04:48:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509524 2024-09-15T04:48:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509524' 2024-09-15T04:48:31.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:31.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:48:31.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346002 2024-09-15T04:48:31.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346002 2024-09-15T04:48:31.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509524 3-85899346002' 2024-09-15T04:48:31.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:31.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:48:31.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182479 2024-09-15T04:48:31.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182479 2024-09-15T04:48:31.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836570 1-42949673047 2-64424509524 3-85899346002 4-107374182479' 2024-09-15T04:48:31.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:31.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836570 2024-09-15T04:48:31.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:31.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:48:31.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836570 2024-09-15T04:48:31.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:31.749 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836570 2024-09-15T04:48:31.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836570 2024-09-15T04:48:31.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836570' 2024-09-15T04:48:31.749 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:32.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836568 -lt 21474836570 2024-09-15T04:48:32.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:48:33.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:48:33.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:33.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836570 -lt 21474836570 2024-09-15T04:48:33.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:33.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673047 2024-09-15T04:48:33.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:33.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:48:33.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673047 2024-09-15T04:48:33.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:33.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673047 2024-09-15T04:48:33.462 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673047 2024-09-15T04:48:33.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673047' 2024-09-15T04:48:33.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:48:33.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673047 -lt 42949673047 2024-09-15T04:48:33.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:33.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509524 2024-09-15T04:48:33.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:33.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:48:33.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509524 2024-09-15T04:48:33.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:33.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509524 2024-09-15T04:48:33.818 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509524 2024-09-15T04:48:33.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509524' 2024-09-15T04:48:33.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:48:34.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509524 -lt 64424509524 2024-09-15T04:48:34.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:34.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346002 2024-09-15T04:48:34.178 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:48:34.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346002 2024-09-15T04:48:34.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:34.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346002 2024-09-15T04:48:34.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346002' 2024-09-15T04:48:34.181 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899346002 2024-09-15T04:48:34.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:48:34.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346002 -lt 85899346002 2024-09-15T04:48:34.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:34.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182479 2024-09-15T04:48:34.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:34.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:48:34.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182479 2024-09-15T04:48:34.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:34.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182479 2024-09-15T04:48:34.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182479' 2024-09-15T04:48:34.574 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182479 2024-09-15T04:48:34.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:48:34.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182480 -lt 107374182479 2024-09-15T04:48:34.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:34.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:34.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:34.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:35.008 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:35.009 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:35.010 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:35.011 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:35.011 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:35.011 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:35.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-15T04:48:24.038159+0000 95'600 107:677 [1,4] 1 [1] 1 0'0 2024-09-15T04:42:54.403964+0000 0'0 2024-09-15T04:42:54.403964+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:39:21.915386+0000 0 0 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:43:23.991206+0000 95'600 107:678 [1] 1 [1] 1 0'0 2024-09-15T04:42:42.819245+0000 0'0 2024-09-15T04:42:42.819245+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:42:02.515009+0000 0 0 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-15T04:48:30.412170+0000 95'600 108:684 [1,2] 1 [1] 1 0'0 2024-09-15T04:42:37.950625+0000 0'0 2024-09-15T04:42:37.950625+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:54:24.081400+0000 0 0 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:48:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:48:35.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-09-15T04:48:35.947 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 3 size to 2 2024-09-15T04:48:35.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-09-15T04:48:37.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-09-15T04:48:37.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:48:37.959 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:48:38.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:48:38.323 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:38.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:48:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836573 2024-09-15T04:48:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836573 2024-09-15T04:48:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573' 2024-09-15T04:48:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:38.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:48:38.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673050 2024-09-15T04:48:38.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673050 2024-09-15T04:48:38.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050' 2024-09-15T04:48:38.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:38.588 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:48:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509528 2024-09-15T04:48:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509528 2024-09-15T04:48:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528' 2024-09-15T04:48:38.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:38.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:48:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346005 2024-09-15T04:48:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346005 2024-09-15T04:48:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528 3-85899346005' 2024-09-15T04:48:38.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:38.805 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:48:38.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182483 2024-09-15T04:48:38.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182483 2024-09-15T04:48:38.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836573 1-42949673050 2-64424509528 3-85899346005 4-107374182483' 2024-09-15T04:48:38.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:38.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836573 2024-09-15T04:48:38.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:38.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:48:38.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836573 2024-09-15T04:48:38.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:38.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836573 2024-09-15T04:48:38.926 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836573 2024-09-15T04:48:38.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836573' 2024-09-15T04:48:38.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:39.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836571 -lt 21474836573 2024-09-15T04:48:39.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:48:40.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:48:40.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:40.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836574 -lt 21474836573 2024-09-15T04:48:40.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:40.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673050 2024-09-15T04:48:40.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:40.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:48:40.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673050 2024-09-15T04:48:40.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:40.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673050 2024-09-15T04:48:40.640 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673050 2024-09-15T04:48:40.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673050' 2024-09-15T04:48:40.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:48:41.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673050 -lt 42949673050 2024-09-15T04:48:41.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:41.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509528 2024-09-15T04:48:41.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:41.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:48:41.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509528 2024-09-15T04:48:41.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:41.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509528 2024-09-15T04:48:41.021 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509528 2024-09-15T04:48:41.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509528' 2024-09-15T04:48:41.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:48:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509528 -lt 64424509528 2024-09-15T04:48:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:41.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346005 2024-09-15T04:48:41.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:41.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:48:41.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346005 2024-09-15T04:48:41.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:41.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346005 2024-09-15T04:48:41.407 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899346005 2024-09-15T04:48:41.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346005' 2024-09-15T04:48:41.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:48:41.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346006 -lt 85899346005 2024-09-15T04:48:41.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:41.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182483 2024-09-15T04:48:41.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:41.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:48:41.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182483 2024-09-15T04:48:41.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:41.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182483 2024-09-15T04:48:41.769 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182483 2024-09-15T04:48:41.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182483' 2024-09-15T04:48:41.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:48:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182483 -lt 107374182483 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:42.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:42.203 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:48:42.204 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:42.205 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:42.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-09-15T04:48:24.038159+0000 95'600 112:681 [1,4] 1 [1] 1 0'0 2024-09-15T04:42:54.403964+0000 0'0 2024-09-15T04:42:54.403964+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:39:21.915386+0000 0 0 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-15T04:48:37.392281+0000 95'600 113:691 [1,2] 1 [1] 1 0'0 2024-09-15T04:42:42.819245+0000 0'0 2024-09-15T04:42:42.819245+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:57:43.238357+0000 0 0 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-09-15T04:48:30.412170+0000 95'600 112:688 [1,2] 1 [1] 1 0'0 2024-09-15T04:42:37.950625+0000 0'0 2024-09-15T04:42:37.950625+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:54:24.081400+0000 0 0 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:48:42.631 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:48:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:48:42.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:42.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-09-15T04:48:42.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-09-15T04:48:42.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-15T04:48:42.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:42.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-15T04:48:42.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-15T04:48:42.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-09-15T04:48:42.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-15T04:48:42.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:42.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-15T04:48:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-09-15T04:48:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-15T04:48:42.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-09-15T04:48:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-09-15T04:48:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-09-15T04:48:42.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-09-15T04:48:42.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-09-15T04:48:42.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:44.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:45.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:45.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:45.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:48:45.091 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 255, 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:45.092 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:45.093 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:45.094 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:45.094 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:45.094 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:45.094 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:48:45.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:48:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:45.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:48:45.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836577 2024-09-15T04:48:45.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836577 2024-09-15T04:48:45.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836577' 2024-09-15T04:48:45.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:45.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:48:45.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673053 2024-09-15T04:48:45.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673053 2024-09-15T04:48:45.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836577 1-42949673053' 2024-09-15T04:48:45.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:45.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:48:45.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509531 2024-09-15T04:48:45.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509531 2024-09-15T04:48:45.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509531' 2024-09-15T04:48:45.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:45.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:48:45.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346009 2024-09-15T04:48:45.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346009 2024-09-15T04:48:45.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509531 3-85899346009' 2024-09-15T04:48:45.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:45.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:48:45.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182486 2024-09-15T04:48:45.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182486 2024-09-15T04:48:45.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836577 1-42949673053 2-64424509531 3-85899346009 4-107374182486' 2024-09-15T04:48:45.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:45.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836577 2024-09-15T04:48:45.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:45.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:48:45.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836577 2024-09-15T04:48:45.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:45.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836577 2024-09-15T04:48:45.996 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836577 2024-09-15T04:48:45.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836577' 2024-09-15T04:48:45.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:46.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836574 -lt 21474836577 2024-09-15T04:48:46.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:48:47.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:48:47.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:47.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836577 -lt 21474836577 2024-09-15T04:48:47.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:47.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673053 2024-09-15T04:48:47.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:47.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:48:47.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673053 2024-09-15T04:48:47.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673053 2024-09-15T04:48:47.709 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673053 2024-09-15T04:48:47.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673053' 2024-09-15T04:48:47.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:48:48.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673054 -lt 42949673053 2024-09-15T04:48:48.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:48.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509531 2024-09-15T04:48:48.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:48.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:48:48.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509531 2024-09-15T04:48:48.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:48.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509531 2024-09-15T04:48:48.062 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509531 2024-09-15T04:48:48.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509531' 2024-09-15T04:48:48.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:48:48.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509532 -lt 64424509531 2024-09-15T04:48:48.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:48.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346009 2024-09-15T04:48:48.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:48:48.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346009 2024-09-15T04:48:48.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:48.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346009 2024-09-15T04:48:48.419 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899346009 2024-09-15T04:48:48.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346009' 2024-09-15T04:48:48.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:48:48.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346009 -lt 85899346009 2024-09-15T04:48:48.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:48.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182486 2024-09-15T04:48:48.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:48.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:48:48.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182486 2024-09-15T04:48:48.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:48.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182486 2024-09-15T04:48:48.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182486' 2024-09-15T04:48:48.773 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182486 2024-09-15T04:48:48.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:48:49.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182487 -lt 107374182486 2024-09-15T04:48:49.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-09-15T04:48:49.463 INFO:tasks.workunit.client.0.smithi138.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-09-15T04:48:49.463 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T04:48:49.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:51.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:51.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:51.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:51.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:51.568 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:51.568 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:51.568 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:51.568 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:51.569 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:51.570 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:51.571 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:51.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-15T04:48:51.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-09-15T04:48:51.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-09-15T04:48:51.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-09-15T04:48:51.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:48:51.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-15T04:48:51.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-09-15T04:48:51.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-15T04:48:51.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-09-15T04:48:51.910 INFO:tasks.workunit.client.0.smithi138.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-09-15T04:48:51.910 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T04:48:51.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:48:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:57.005 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:48:57.006 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:48:57.007 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:48:57.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-09-15T04:48:57.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:48:57.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:57.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:48:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836581 2024-09-15T04:48:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836581 2024-09-15T04:48:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581' 2024-09-15T04:48:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:57.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:48:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673058 2024-09-15T04:48:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673058 2024-09-15T04:48:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673058' 2024-09-15T04:48:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:57.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:48:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509535 2024-09-15T04:48:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509535 2024-09-15T04:48:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673058 2-64424509535' 2024-09-15T04:48:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:57.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:48:57.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346013 2024-09-15T04:48:57.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346013 2024-09-15T04:48:57.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673058 2-64424509535 3-85899346013' 2024-09-15T04:48:57.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:48:57.807 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:48:57.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182491 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182491 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836581 1-42949673058 2-64424509535 3-85899346013 4-107374182491' 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836581 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:57.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:48:57.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836581 2024-09-15T04:48:57.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:57.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836581 2024-09-15T04:48:57.916 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836581 2024-09-15T04:48:57.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836581' 2024-09-15T04:48:57.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:58.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836579 -lt 21474836581 2024-09-15T04:48:58.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:48:59.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:48:59.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:48:59.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836581 -lt 21474836581 2024-09-15T04:48:59.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:59.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673058 2024-09-15T04:48:59.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:59.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:48:59.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673058 2024-09-15T04:48:59.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673058 2024-09-15T04:48:59.635 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673058 2024-09-15T04:48:59.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673058' 2024-09-15T04:48:59.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:48:59.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673058 -lt 42949673058 2024-09-15T04:48:59.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:48:59.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509535 2024-09-15T04:48:59.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:48:59.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:48:59.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509535 2024-09-15T04:48:59.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:48:59.994 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509535 2024-09-15T04:48:59.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509535 2024-09-15T04:48:59.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509535' 2024-09-15T04:48:59.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:49:00.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509536 -lt 64424509535 2024-09-15T04:49:00.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:00.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346013 2024-09-15T04:49:00.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:00.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:49:00.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346013 2024-09-15T04:49:00.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:00.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346013 2024-09-15T04:49:00.348 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899346013 2024-09-15T04:49:00.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346013' 2024-09-15T04:49:00.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:49:00.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346014 -lt 85899346013 2024-09-15T04:49:00.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:00.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182491 2024-09-15T04:49:00.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:00.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:49:00.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182491 2024-09-15T04:49:00.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:00.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182491 2024-09-15T04:49:00.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182491' 2024-09-15T04:49:00.707 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182491 2024-09-15T04:49:00.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:49:01.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182491 -lt 107374182491 2024-09-15T04:49:01.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-09-15T04:49:01.438 INFO:tasks.workunit.client.0.smithi138.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-09-15T04:49:01.439 INFO:tasks.workunit.client.0.smithi138.stderr: 2024-09-15T04:49:01.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-09-15T04:49:03.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:49:03.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:49:03.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:49:03.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:03.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:49:03.542 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:03.542 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:03.542 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:03.542 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:03.542 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:03.543 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:03.544 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:03.545 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:03.545 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:03.545 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:03.545 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:03.545 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:03.546 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:03.547 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:03.548 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:03.548 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:03.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:49:03.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-09-15T04:49:03.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-09-15T04:49:03.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-09-15T04:49:03.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:49:03.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-09-15T04:49:03.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-09-15T04:49:03.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-09-15T04:49:03.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-09-15T04:49:03.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-09-15T04:49:03.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-09-15T04:49:03.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-09-15T04:49:03.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-09-15T04:49:03.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-09-15T04:49:04.173 INFO:tasks.workunit.client.0.smithi138.stderr:noout is unset 2024-09-15T04:49:04.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-09-15T04:49:04.853 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations' 2024-09-15T04:49:04.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations' 2024-09-15T04:49:04.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:49:04.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:49:04.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:49:04.866 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:49:04.867 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:49:04.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:49:04.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:49:04.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:49:04.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:49:04.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:49:04.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:49:04.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:49:04.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:49:04.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:49:05.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:49:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:49:05.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:49:05.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836585 2024-09-15T04:49:05.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836585 2024-09-15T04:49:05.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585' 2024-09-15T04:49:05.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:49:05.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:49:05.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673061 2024-09-15T04:49:05.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673061 2024-09-15T04:49:05.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673061' 2024-09-15T04:49:05.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:49:05.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T04:49:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509539 2024-09-15T04:49:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509539 2024-09-15T04:49:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509539' 2024-09-15T04:49:06.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:49:06.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T04:49:06.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899346017 2024-09-15T04:49:06.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899346017 2024-09-15T04:49:06.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509539 3-85899346017' 2024-09-15T04:49:06.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:49:06.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T04:49:06.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182494 2024-09-15T04:49:06.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182494 2024-09-15T04:49:06.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836585 1-42949673061 2-64424509539 3-85899346017 4-107374182494' 2024-09-15T04:49:06.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:06.320 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836585 2024-09-15T04:49:06.320 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:06.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:49:06.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836585 2024-09-15T04:49:06.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:06.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836585 2024-09-15T04:49:06.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836585' 2024-09-15T04:49:06.322 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836585 2024-09-15T04:49:06.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:49:06.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836585 -lt 21474836585 2024-09-15T04:49:06.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:06.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673061 2024-09-15T04:49:06.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:06.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:49:06.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673061 2024-09-15T04:49:06.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:06.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673061 2024-09-15T04:49:06.792 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673061 2024-09-15T04:49:06.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673061' 2024-09-15T04:49:06.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:49:07.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673062 -lt 42949673061 2024-09-15T04:49:07.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:07.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509539 2024-09-15T04:49:07.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:07.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T04:49:07.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509539 2024-09-15T04:49:07.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:07.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509539 2024-09-15T04:49:07.429 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509539 2024-09-15T04:49:07.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509539' 2024-09-15T04:49:07.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T04:49:07.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509539 -lt 64424509539 2024-09-15T04:49:07.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:07.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899346017 2024-09-15T04:49:07.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:07.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T04:49:07.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899346017 2024-09-15T04:49:07.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:07.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899346017 2024-09-15T04:49:07.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899346017' 2024-09-15T04:49:07.990 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899346017 2024-09-15T04:49:07.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T04:49:08.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899346017 -lt 85899346017 2024-09-15T04:49:08.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:49:08.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182494 2024-09-15T04:49:08.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:49:08.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T04:49:08.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182494 2024-09-15T04:49:08.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:49:08.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182494 2024-09-15T04:49:08.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182494' 2024-09-15T04:49:08.468 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182494 2024-09-15T04:49:08.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T04:49:08.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182495 -lt 107374182494 2024-09-15T04:49:08.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:49:08.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:08.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:09.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:09.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:09.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:09.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:09.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:09.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:10.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:10.540 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:10.540 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:10.541 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:10.542 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:10.543 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:10.544 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:10.545 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:10.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:10.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:10.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:10.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:10.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:10.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:10.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:10.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:10.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:11.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:11.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:11.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:11.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:12.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3447205 2024-09-15T04:49:12.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3447205 '!=' null 2024-09-15T04:49:12.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:12.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:12.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:12.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:12.091 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:12.091 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:12.092 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:12.092 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:12.092 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:12.092 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:12.092 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:12.093 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:12.094 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:12.095 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:12.096 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:12.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:12.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:12.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:12.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:12.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:12.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:12.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:13.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:13.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14415650 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14415650 '!=' null 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:13.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:13.731 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:13.731 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:13.731 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:13.731 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:13.732 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:13.733 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:13.734 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:13.734 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:13.734 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:13.734 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:13.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:13.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:13.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:13.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:14.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:14.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:14.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:14.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:14.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:15.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11965885 2024-09-15T04:49:15.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11965885 '!=' null 2024-09-15T04:49:15.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:15.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:15.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:15.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:15.221 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:15.221 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:15.221 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:15.221 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:15.222 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:15.223 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:15.224 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:15.224 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:15.224 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:15.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:15.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:15.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:15.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:15.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:15.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:15.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:15.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:15.331 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:15.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:15.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:15.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:15.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:16.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:16.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:16.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:16.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18388575 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18388575 '!=' null 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:16.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:16.785 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:16.785 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:16.785 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:16.785 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:16.785 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:16.786 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:16.787 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:16.788 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:16.788 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:16.788 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:16.788 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:16.788 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:16.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:16.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:16.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:17.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:17.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:17.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:17.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:17.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18388575 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18388575 '!=' null 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:18.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:18.396 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:18.397 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:18.398 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:18.399 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:18.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:18.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:18.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:18.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:18.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:18.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:18.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:18.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:19.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:19.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726018 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726018 '!=' null 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:20.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:20.120 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:20.121 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:20.122 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:20.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:20.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:20.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:20.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:20.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:20.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:21.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:21.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726043 2024-09-15T04:49:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726043 '!=' null 2024-09-15T04:49:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:21.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:21.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:21.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:21.811 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:21.811 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:21.811 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:21.811 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:21.812 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:21.813 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:21.814 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:21.814 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:21.814 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:21.814 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:21.814 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:21.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:21.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:21.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:22.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:22.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:22.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:22.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:22.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:22.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:22.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:22.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:22.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:22.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:22.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:22.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:22.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462732 2024-09-15T04:49:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462732 '!=' null 2024-09-15T04:49:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:23.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:23.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:23.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:23.477 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:23.477 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:23.477 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:23.477 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:23.477 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:23.478 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:23.479 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:23.480 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:23.480 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:23.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:23.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:23.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:23.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:23.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:23.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:24.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:24.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:25.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599724 2024-09-15T04:49:25.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599724 '!=' null 2024-09-15T04:49:25.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:25.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:25.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:25.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:25.169 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:25.170 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:25.171 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:25.172 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:25.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:25.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:25.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:25.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:25.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:25.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:25.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:26.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:26.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:26.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:26.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:26.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:26.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336501 2024-09-15T04:49:26.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336501 '!=' null 2024-09-15T04:49:26.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:26.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:26.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:26.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:26.939 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:26.939 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:26.939 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:26.939 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:26.939 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:26.940 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:26.941 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:26.942 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:26.943 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:26.943 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:26.943 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:26.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:27.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:27.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:27.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:27.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:27.486 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:27.486 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:27.486 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:27.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:27.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:28.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336501 2024-09-15T04:49:28.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336501 '!=' null 2024-09-15T04:49:28.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:28.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:28.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:28.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:28.508 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:28.509 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:28.510 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:28.511 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:28.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:28.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:28.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:28.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:28.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:28.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:28.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:29.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:29.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:30.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473288 2024-09-15T04:49:30.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473288 '!=' null 2024-09-15T04:49:30.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:30.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:30.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:30.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:30.232 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:30.232 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:30.232 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:30.232 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:30.233 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:30.234 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:30.235 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:30.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:30.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:30.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:30.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:30.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:30.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:30.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:30.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:30.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:30.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:30.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:31.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:31.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:31.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:31.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:31.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:31.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:31.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:31.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:31.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:31.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473403 2024-09-15T04:49:31.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473403 '!=' null 2024-09-15T04:49:31.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:31.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:31.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:31.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:31.934 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:31.934 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:31.934 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:31.934 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:31.934 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:31.935 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:31.936 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:31.937 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:31.937 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:31.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:32.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:32.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:32.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:32.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:32.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:32.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:33.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26210109 2024-09-15T04:49:33.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26210109 '!=' null 2024-09-15T04:49:33.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:33.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:33.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:33.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:33.446 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:33.447 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:33.448 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:33.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:33.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:33.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:33.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:33.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:33.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:33.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:33.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:33.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:33.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:33.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:33.905 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:33.905 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:34.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:34.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:34.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:34.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:35.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473284 2024-09-15T04:49:35.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473284 '!=' null 2024-09-15T04:49:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:35.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:35.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:35.122 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:35.123 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:35.124 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:35.125 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:35.125 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:35.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:35.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:35.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:35.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:35.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:35.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:35.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:35.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:36.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:36.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:36.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209870 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209870 '!=' null 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:36.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:36.690 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:36.691 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:36.692 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:36.693 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:36.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:36.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:36.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:37.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:37.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:37.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:37.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:37.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:37.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:37.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:37.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:37.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:37.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:37.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:37.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:37.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209870 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209870 '!=' null 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:38.167 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:38.239 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:38.239 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:38.240 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:38.241 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:38.241 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:38.241 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:38.241 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:38.241 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:38.242 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:38.243 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:38.244 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:38.245 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:38.245 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:38.245 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:38.245 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:38.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:38.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:38.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:38.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:38.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:38.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:38.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:38.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:38.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:38.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:38.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:38.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:38.695 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:39.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473225 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473225 '!=' null 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:39.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:39.889 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:39.890 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:39.891 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:39.892 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:39.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:39.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:39.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:39.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:39.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:39.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:39.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:39.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:40.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:40.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:40.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:40.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:40.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:40.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:40.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17473309 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17473309 '!=' null 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:41.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:41.713 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:41.714 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:41.715 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:41.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:41.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:42.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:42.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:42.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:42.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:42.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:42.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336434 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336434 '!=' null 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:43.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:43.294 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:43.294 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:43.294 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:43.294 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:43.294 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:43.295 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:43.296 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:43.297 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:43.297 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:43.297 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:43.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:43.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:43.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:43.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:43.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:43.404 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:43.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:43.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:43.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:43.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:43.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:43.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:43.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:44.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:44.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:44.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:44.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599496 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599496 '!=' null 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:44.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:45.043 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:45.069 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:45.069 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:45.069 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:45.069 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:45.070 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:45.070 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:45.070 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:45.070 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:45.070 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:45.071 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:45.072 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:45.073 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:45.074 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:45.074 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:45.074 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:45.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:45.075 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:45.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:45.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:45.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:45.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:45.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:45.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:45.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:45.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462493 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462493 '!=' null 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:46.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:46.691 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:46.692 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:46.693 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:46.694 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:46.694 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:46.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:46.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:46.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:46.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:47.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:47.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:47.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:47.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:47.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:47.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:48.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462493 2024-09-15T04:49:48.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462493 '!=' null 2024-09-15T04:49:48.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:48.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:48.219 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:48.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:48.292 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:48.293 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:48.294 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:48.295 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:48.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:48.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:48.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:48.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:48.744 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:48.744 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:48.744 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:49.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:49.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:49.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725878 2024-09-15T04:49:49.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725878 '!=' null 2024-09-15T04:49:49.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:49.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:49.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:49.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:49.993 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:49.994 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:49.995 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:49.996 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:50.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:50.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:50.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:50.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:50.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:50.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:50.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:50.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725904 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725904 '!=' null 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:51.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:51.493 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:51.494 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:51.495 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:51.496 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:51.496 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:51.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:51.606 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:51.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:51.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:51.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:51.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:52.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:52.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:52.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:52.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:53.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588724 2024-09-15T04:49:53.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588724 '!=' null 2024-09-15T04:49:53.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:53.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:53.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:53.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:53.149 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:53.150 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:53.151 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:53.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:53.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:53.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:53.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:53.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:53.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:53.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:53.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:53.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:53.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:53.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:53.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:53.604 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:54.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:54.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:54.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:54.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725667 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725667 '!=' null 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:54.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:54.781 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:54.782 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:54.783 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:54.784 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:54.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:54.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:54.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:54.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:54.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:54.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:54.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:54.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:54.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:55.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:55.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:55.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:55.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:55.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:55.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:56.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588750 2024-09-15T04:49:56.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588750 '!=' null 2024-09-15T04:49:56.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:56.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:56.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:56.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:56.602 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:56.603 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:56.604 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:56.605 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:56.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:56.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:56.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:56.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:57.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:57.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588750 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588750 '!=' null 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:58.180 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:58.252 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:49:58.252 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:49:58.252 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:58.252 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:49:58.253 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:49:58.254 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:49:58.255 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:49:58.255 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:49:58.255 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:49:58.255 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:49:58.255 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:49:58.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:49:58.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:49:58.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:49:58.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:49:58.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:49:58.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:49:59.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725775 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725775 '!=' null 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:49:59.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:00.050 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:00.051 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:00.052 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:00.053 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:00.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:00.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:00.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:00.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:00.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:00.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:00.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:00.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:01.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:01.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:01.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725742 2024-09-15T04:50:01.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725742 '!=' null 2024-09-15T04:50:01.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:01.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:01.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:01.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:01.721 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:01.722 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:01.723 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:01.724 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:01.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:01.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:02.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:02.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:02.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:02.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:02.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:02.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:02.839 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:03.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588560 2024-09-15T04:50:03.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588560 '!=' null 2024-09-15T04:50:03.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:03.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:03.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:03.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:03.594 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:03.595 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:03.596 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:03.597 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:03.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:03.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:04.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:04.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:04.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:04.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:04.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:04.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:05.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725819 2024-09-15T04:50:05.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725819 '!=' null 2024-09-15T04:50:05.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:05.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:05.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:05.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:05.181 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:05.181 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:05.182 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:05.183 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:05.184 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:05.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:05.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:05.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:05.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:05.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:05.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:05.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:05.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:05.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:05.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:05.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:05.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:05.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:06.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:06.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:06.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:06.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:06.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462585 2024-09-15T04:50:06.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462585 '!=' null 2024-09-15T04:50:06.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:06.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:06.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:06.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:06.743 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:06.744 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:06.746 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:06.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:06.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:06.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:06.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:07.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:07.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:07.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:07.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:07.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:07.679 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:08.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462585 2024-09-15T04:50:08.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462585 '!=' null 2024-09-15T04:50:08.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:08.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:08.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:08.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:08.301 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:08.301 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:08.301 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:08.302 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:08.303 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:08.304 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:08.304 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:08.304 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:08.304 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:08.304 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:08.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:08.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:08.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:08.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:08.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:08.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:08.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:09.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599536 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599536 '!=' null 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:09.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:10.057 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:10.058 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:10.059 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:10.060 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:10.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:10.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:10.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:10.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:10.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:10.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:10.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:10.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:11.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:11.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:11.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599563 2024-09-15T04:50:11.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599563 '!=' null 2024-09-15T04:50:11.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:11.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:11.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:11.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:11.657 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:11.657 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:11.657 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:11.658 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:11.659 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:11.660 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:11.660 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:11.660 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:11.660 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:11.660 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:11.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:11.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:11.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:11.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:12.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:12.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:12.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:12.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:12.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:12.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:12.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:12.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:13.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462569 2024-09-15T04:50:13.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462569 '!=' null 2024-09-15T04:50:13.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:13.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:13.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:13.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:13.235 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:13.237 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:13.237 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:13.237 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:13.237 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:13.237 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:13.238 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:13.238 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:13.238 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:13.238 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:13.238 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:13.239 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:13.240 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:13.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:13.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:13.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:13.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:13.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:13.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:13.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:14.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:14.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:14.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:14.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:14.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:14.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:14.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599594 2024-09-15T04:50:14.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599594 '!=' null 2024-09-15T04:50:14.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:14.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:14.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:14.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:14.963 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:14.963 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:14.963 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:14.964 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:14.965 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:14.966 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:14.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:15.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:15.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:15.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:15.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:15.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:15.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:15.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:15.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:15.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599594 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599594 '!=' null 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:16.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:16.553 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:16.556 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:16.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:16.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:16.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:16.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:17.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:17.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:17.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:17.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:17.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462549 2024-09-15T04:50:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462549 '!=' null 2024-09-15T04:50:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:18.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:18.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:18.197 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:18.197 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:18.197 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:18.197 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:18.197 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:18.198 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:18.199 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:18.200 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:18.200 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:18.200 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:18.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:18.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:18.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:18.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:18.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:18.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:19.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:19.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:19.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725716 2024-09-15T04:50:19.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725716 '!=' null 2024-09-15T04:50:19.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:19.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:19.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:19.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:19.862 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:19.863 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:19.864 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:19.865 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:19.866 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:19.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:19.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:19.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:19.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:19.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:19.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:19.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:19.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:19.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:20.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:20.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:20.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:20.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:20.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:20.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:20.819 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725537 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725537 '!=' null 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:21.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:21.525 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:21.525 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:21.526 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:21.527 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:21.527 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:21.528 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:21.529 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:21.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:21.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:21.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:21.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:21.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:21.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:21.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:21.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:21.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:21.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:21.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:21.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:21.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:22.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:22.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:22.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588226 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588226 '!=' null 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:23.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:23.074 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:23.075 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:23.076 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:23.077 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:23.077 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:23.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:23.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:23.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:23.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:23.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:23.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:24.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:24.105 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:24.105 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:24.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725485 2024-09-15T04:50:24.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725485 '!=' null 2024-09-15T04:50:24.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:24.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:24.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:24.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:24.805 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:24.806 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:24.807 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:24.808 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:24.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:24.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:24.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:24.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:25.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:25.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:25.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:25.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:25.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:25.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:26.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725485 2024-09-15T04:50:26.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725485 '!=' null 2024-09-15T04:50:26.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:26.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:26.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:26.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:26.407 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:26.408 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:26.409 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:26.410 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:26.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:26.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:26.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:26.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:26.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:26.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:26.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:26.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:26.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:26.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:26.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:26.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:26.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:27.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461946 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461946 '!=' null 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:27.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:28.060 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:28.061 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:28.063 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:28.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:28.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:28.170 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:28.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:28.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:28.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:28.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:29.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:29.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:29.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:29.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:29.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:29.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:29.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:29.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:29.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:29.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598944 2024-09-15T04:50:29.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598944 '!=' null 2024-09-15T04:50:29.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:29.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:29.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:29.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:29.732 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:29.732 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:29.732 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:29.733 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:29.734 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:29.735 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:29.735 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:29.735 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:29.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:29.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:29.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:29.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:29.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:29.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:29.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:29.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:29.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:30.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:30.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:30.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:30.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:30.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:30.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:30.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:30.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598917 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598917 '!=' null 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:31.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:31.470 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:31.471 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:31.472 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:31.473 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:31.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:31.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:31.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:31.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:31.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:31.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:31.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:31.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:31.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:31.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:31.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:31.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:31.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:32.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:33.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588304 2024-09-15T04:50:33.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588304 '!=' null 2024-09-15T04:50:33.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:33.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:33.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:33.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:33.309 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:33.310 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:33.311 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:33.312 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:33.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:33.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:33.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:33.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:33.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:33.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:35.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725609 2024-09-15T04:50:35.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725609 '!=' null 2024-09-15T04:50:35.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:35.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:35.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:35.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:35.141 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:35.142 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:35.143 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:35.144 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:35.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:35.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:35.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:35.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:35.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:35.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:35.714 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:35.714 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:36.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:36.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:36.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461882 2024-09-15T04:50:36.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461882 '!=' null 2024-09-15T04:50:36.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:36.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:36.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:36.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:36.953 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:36.953 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:36.954 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:36.955 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:36.956 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:36.956 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:36.956 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:36.956 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:36.956 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:36.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:37.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:37.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:37.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:37.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:37.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:37.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:37.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461882 2024-09-15T04:50:38.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461882 '!=' null 2024-09-15T04:50:38.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:38.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:38.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:38.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:38.550 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:38.551 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:38.552 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:38.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:38.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:38.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:38.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:39.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:39.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:39.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:39.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:39.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:39.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:39.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:39.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:39.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:39.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:39.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:39.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:39.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:40.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725425 2024-09-15T04:50:40.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725425 '!=' null 2024-09-15T04:50:40.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:40.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:40.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:40.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:40.265 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:40.266 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:40.267 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:40.268 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:40.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:40.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:40.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:40.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:40.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:40.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:41.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:41.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725636 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725636 '!=' null 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:41.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:41.844 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:41.845 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:41.846 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:41.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:41.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:41.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:41.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:42.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:42.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:42.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:42.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:42.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:42.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:42.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:43.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588759 2024-09-15T04:50:43.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588759 '!=' null 2024-09-15T04:50:43.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:43.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:43.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:43.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:43.466 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:43.467 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:43.468 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:43.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:43.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:43.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:43.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:43.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:43.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:43.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:43.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:43.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:43.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:43.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:43.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:43.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:44.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:44.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599499 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599499 '!=' null 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:45.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:45.226 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:45.227 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:45.228 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:45.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:45.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:45.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:45.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:45.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:45.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:45.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:45.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:45.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:45.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:45.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:45.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:45.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:46.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:46.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:46.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:46.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:46.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462210 2024-09-15T04:50:46.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462210 '!=' null 2024-09-15T04:50:46.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:46.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:46.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:46.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:46.807 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:46.808 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:46.809 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:46.810 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:46.810 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:46.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:46.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:46.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:47.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:47.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:47.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:47.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:47.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:47.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:47.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:47.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:47.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:47.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:47.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:47.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:47.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725478 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725478 '!=' null 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:48.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:48.634 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:48.635 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:48.636 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:48.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:48.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:48.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:48.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:49.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:49.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:49.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:49.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:49.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725478 2024-09-15T04:50:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725478 '!=' null 2024-09-15T04:50:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:50.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:50.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:50.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:50.207 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:50.208 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:50.209 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:50.210 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:50.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:50.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:50.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:50.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:50.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:50.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:50.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:50.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:50.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:50.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:50.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:50.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:50.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:51.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:51.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725528 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725528 '!=' null 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:51.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:51.790 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:51.790 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:51.790 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:51.790 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:51.790 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:51.791 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "7.0", 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 255, 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:51.792 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:51.793 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:51.793 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:51.793 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:51.793 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:51.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:51.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:51.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:51.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:52.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:52.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:52.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:52.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:53.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:53.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:53.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:53.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19918946 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19918946 '!=' null 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:54.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:54.077 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:54.078 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:54.079 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:54.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:54.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:54.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:54.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:54.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:54.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:54.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:54.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:54.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:54.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:54.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:54.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:54.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:55.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:55.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:55.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:55.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:55.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:55.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:55.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:55.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:55.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104705 2024-09-15T04:50:55.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104705 '!=' null 2024-09-15T04:50:55.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:55.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:55.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:55.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:55.801 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:55.801 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:55.801 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:55.801 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:55.802 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:55.803 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:55.804 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:55.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:55.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:56.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T04:50:56.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:56.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:56.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:56.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-09-15T04:50:56.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T04:50:56.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:56.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:56.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:56.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:56.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:56.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:56.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:50:57.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27520769 2024-09-15T04:50:57.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27520769 '!=' null 2024-09-15T04:50:57.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:50:57.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:57.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:57.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:57.362 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:57.362 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:57.362 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:57.362 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:57.362 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:57.363 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:57.364 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:57.365 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:57.365 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:57.365 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:57.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:57.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:57.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:50:57.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:57.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:57.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:50:58.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:50:58.450 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:50:58.450 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:50:58.450 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:58.451 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:50:58.452 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:50:58.453 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:50:58.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:50:58.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:50:58.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:50:58.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:50:58.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:50:58.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:50:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:50:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:50:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:50:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:50:59.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:50:59.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:50:59.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:50:59.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:50:59.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:00.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27520484 2024-09-15T04:51:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27520484 '!=' null 2024-09-15T04:51:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:00.104 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:00.104 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:00.177 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:00.178 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:00.179 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:00.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:00.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:00.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:00.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:00.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:00.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:01.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:01.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:01.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14263647 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14263647 '!=' null 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:01.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:01.779 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:01.779 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:01.779 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:01.780 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:01.781 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:01.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:01.888 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:01.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:01.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:02.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:02.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:02.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:02.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:02.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:02.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:02.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:03.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22016271 2024-09-15T04:51:03.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22016271 '!=' null 2024-09-15T04:51:03.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:03.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:03.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:03.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:03.574 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:03.575 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:03.576 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:03.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:03.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:03.684 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:03.684 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:04.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:04.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:04.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:04.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:04.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:04.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:04.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:05.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19928321 2024-09-15T04:51:05.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19928321 '!=' null 2024-09-15T04:51:05.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:05.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:05.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:05.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:05.229 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:05.230 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:05.231 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:05.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:05.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:05.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:05.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:05.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:05.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:05.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:05.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:06.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:06.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:06.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:06.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209819 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209819 '!=' null 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:06.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:06.886 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:06.886 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:06.887 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:06.888 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:06.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:06.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:06.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:06.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:07.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:07.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:07.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:07.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:07.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26209819 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26209819 '!=' null 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:08.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:08.372 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:08.373 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:08.374 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:08.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:08.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:08.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:08.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:08.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:08.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:08.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:09.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:09.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:09.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599520 2024-09-15T04:51:09.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599520 '!=' null 2024-09-15T04:51:09.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:09.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:09.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:09.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:09.980 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:09.981 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:09.982 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:09.983 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:09.983 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:09.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:10.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:10.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:10.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:10.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:10.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:10.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:10.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:10.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:10.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:10.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:10.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:10.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:10.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:10.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599678 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599678 '!=' null 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:11.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:11.501 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:11.501 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:11.502 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:11.503 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:11.504 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:11.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:11.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:11.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:11.610 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:12.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:12.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:12.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:12.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:12.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:12.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336217 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336217 '!=' null 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:13.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:13.361 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:13.362 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:13.363 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:13.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:13.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:13.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:13.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:13.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:13.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:13.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:13.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:13.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:13.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:13.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:13.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:13.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:14.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:14.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:14.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599638 2024-09-15T04:51:14.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599638 '!=' null 2024-09-15T04:51:14.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:14.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:14.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:14.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:14.999 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:15.000 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:15.001 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:15.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:15.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:15.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:15.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:15.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:15.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:15.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:15.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599638 2024-09-15T04:51:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599638 '!=' null 2024-09-15T04:51:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:16.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:16.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:16.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:16.554 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:16.554 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:16.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:16.556 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:16.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:16.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:16.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:16.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:17.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:17.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:17.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:17.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:17.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:17.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462691 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462691 '!=' null 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:18.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:18.305 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:18.306 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:18.307 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:18.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:18.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:18.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:18.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:18.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:18.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:18.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:18.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:18.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:18.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:18.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:18.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:19.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:19.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:19.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599750 2024-09-15T04:51:19.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599750 '!=' null 2024-09-15T04:51:19.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:19.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:19.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:19.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:19.841 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:19.841 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:19.841 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:19.841 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:19.841 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:19.842 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:19.843 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:19.844 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:19.844 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:19.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:19.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:19.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:19.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:19.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:19.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:19.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:19.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:19.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:20.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:20.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599776 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599776 '!=' null 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:21.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:21.415 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:21.416 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:21.417 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:21.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:21.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:21.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:21.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:21.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:21.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:21.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:21.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:22.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:22.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336218 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336218 '!=' null 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:23.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:23.175 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:23.176 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:23.177 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:23.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:23.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:23.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:23.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:23.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:23.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:24.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:24.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:24.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:24.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:24.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:24.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:24.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:24.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:24.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:24.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599729 2024-09-15T04:51:24.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599729 '!=' null 2024-09-15T04:51:24.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:24.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:24.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:24.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:24.904 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:24.904 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:24.904 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:24.905 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:24.906 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:24.907 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:24.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:25.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:25.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:25.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:25.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:25.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:25.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:25.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:25.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:25.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:25.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:25.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:25.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:25.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:25.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:25.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:25.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599729 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599729 '!=' null 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:26.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:26.550 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:26.550 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:26.551 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:26.552 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:26.553 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:26.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:26.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:26.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:26.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:26.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:26.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:26.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:26.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:27.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:27.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:27.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:27.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:27.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:27.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:28.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462787 2024-09-15T04:51:28.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462787 '!=' null 2024-09-15T04:51:28.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:28.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:28.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:28.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:28.128 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:28.128 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:28.129 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:28.130 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:28.131 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:28.131 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:28.131 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:28.131 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:28.131 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:28.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:28.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:28.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:28.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:29.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:29.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:29.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:29.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599619 2024-09-15T04:51:29.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599619 '!=' null 2024-09-15T04:51:29.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:29.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:29.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:29.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:29.854 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:29.855 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:29.856 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:29.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:29.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:29.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:29.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:30.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:30.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:30.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:30.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599624 2024-09-15T04:51:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599624 '!=' null 2024-09-15T04:51:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:31.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:31.483 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:31.555 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:31.556 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:31.557 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:31.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:31.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:31.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:31.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:32.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:32.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:32.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:32.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:32.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:32.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:32.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336056 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336056 '!=' null 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:33.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:33.075 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:33.075 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:33.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:33.077 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:33.078 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:33.078 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:33.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:33.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:33.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:33.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:33.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:33.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:33.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:34.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:34.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:34.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:34.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599594 2024-09-15T04:51:34.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599594 '!=' null 2024-09-15T04:51:34.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:34.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:34.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:34.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:34.691 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:34.692 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:34.693 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:34.694 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:34.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:34.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:34.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:34.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:34.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:34.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:34.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:34.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:34.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:35.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:35.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:35.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:35.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:35.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:35.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:35.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599594 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599594 '!=' null 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:36.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:36.287 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:36.287 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:36.287 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:36.288 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:36.289 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:36.290 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:36.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:36.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:36.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:36.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:36.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:36.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:36.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:36.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:37.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:37.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462608 2024-09-15T04:51:37.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462608 '!=' null 2024-09-15T04:51:37.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:37.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:37.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:37.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:37.934 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:37.935 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:37.936 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:37.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:38.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:38.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:38.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:38.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:38.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:38.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:38.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:38.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:38.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:38.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:38.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:38.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:38.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:38.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:38.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:39.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599537 2024-09-15T04:51:39.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599537 '!=' null 2024-09-15T04:51:39.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:39.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:39.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:39.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:39.644 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:39.645 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:39.646 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:39.647 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:39.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:39.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:39.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:39.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:39.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:39.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:39.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:39.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:39.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:40.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:40.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:40.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:40.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:40.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:40.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599594 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599594 '!=' null 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:41.161 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:41.233 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:41.234 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:41.235 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:41.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:41.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:41.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:41.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:41.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:41.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:42.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:42.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:42.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462471 2024-09-15T04:51:42.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462471 '!=' null 2024-09-15T04:51:42.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:42.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:42.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:42.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:42.906 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:42.906 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:42.906 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:42.906 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:42.906 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:42.907 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:42.908 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:42.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:43.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:43.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:43.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:43.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:43.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:43.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:43.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:43.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:43.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:43.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:43.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:43.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:43.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:43.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:43.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:43.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462471 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462471 '!=' null 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:44.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:44.522 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:44.523 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:44.524 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:44.524 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:44.524 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:44.524 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:44.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:44.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:44.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:44.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:44.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:44.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:44.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:44.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:45.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:45.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:45.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:46.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725766 2024-09-15T04:51:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725766 '!=' null 2024-09-15T04:51:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:46.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:46.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:46.202 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:46.203 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:46.204 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:46.205 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:46.205 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:46.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:46.205 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:46.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:46.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:46.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:46.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:46.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:46.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:46.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:46.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:47.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:47.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588694 2024-09-15T04:51:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588694 '!=' null 2024-09-15T04:51:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:47.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:47.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:47.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:47.780 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:47.781 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:47.782 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:47.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:47.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:47.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:47.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:48.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:48.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:48.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:48.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:48.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:48.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:48.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:48.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725726 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725726 '!=' null 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:49.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:49.341 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:49.342 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:49.343 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:49.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:49.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:49.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:49.450 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:49.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:49.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:49.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:49.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:50.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:50.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:50.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725805 2024-09-15T04:51:50.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725805 '!=' null 2024-09-15T04:51:50.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:50.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:50.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:50.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:50.966 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:50.967 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:50.968 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:50.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:51.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:51.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:51.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:51.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:51.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:51.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:51.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:51.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:51.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725805 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725805 '!=' null 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:52.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:52.520 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:52.521 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:52.522 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:52.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:52.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:52.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:52.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:52.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:52.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:52.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:52.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:52.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:52.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:52.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:52.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:52.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:53.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:53.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:53.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:53.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:54.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462401 2024-09-15T04:51:54.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462401 '!=' null 2024-09-15T04:51:54.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:54.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:54.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:54.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:54.472 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:54.473 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:54.474 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:54.475 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:54.475 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:54.475 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:54.475 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:54.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:54.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:54.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:54.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:54.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:54.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:54.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:54.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:54.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:55.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:55.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:55.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:55.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:55.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:55.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:56.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599526 2024-09-15T04:51:56.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599526 '!=' null 2024-09-15T04:51:56.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:56.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:56.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:56.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:56.246 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:56.246 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:56.246 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:56.246 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:56.246 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:56.247 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:56.248 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:56.249 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:56.250 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:56.250 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:56.250 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:56.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:56.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:56.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:56.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:56.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:56.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:56.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:56.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:57.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:57.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:57.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:57.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:58.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462873 2024-09-15T04:51:58.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462873 '!=' null 2024-09-15T04:51:58.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:58.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:58.228 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:58.229 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:58.230 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:58.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:58.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:58.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:58.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:51:58.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:51:58.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:51:58.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:51:58.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:51:59.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:51:59.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:51:59.161 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:51:59.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599585 2024-09-15T04:51:59.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599585 '!=' null 2024-09-15T04:51:59.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:51:59.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:59.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:51:59.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:51:59.749 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:51:59.749 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:51:59.749 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:59.750 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:51:59.751 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:51:59.752 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:51:59.752 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:51:59.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:51:59.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:51:59.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:51:59.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:00.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:00.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:00.206 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:00.206 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:00.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:00.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:00.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599690 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599690 '!=' null 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:01.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:01.352 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:01.353 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:01.354 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:01.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:01.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:01.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:01.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:01.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:02.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:02.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:02.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:02.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462641 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462641 '!=' null 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:03.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:03.109 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:03.109 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:03.109 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:03.109 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:03.109 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:03.110 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:03.111 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:03.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:03.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:03.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:03.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:03.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:03.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:03.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:03.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:03.219 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:03.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:03.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:03.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:03.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:04.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:04.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:04.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:04.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:04.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:04.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:04.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:04.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:04.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:04.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726019 2024-09-15T04:52:04.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726019 '!=' null 2024-09-15T04:52:04.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:04.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:04.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:04.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:04.928 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:04.928 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:04.928 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:04.929 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:04.930 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:04.931 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:04.931 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:04.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:05.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:05.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:05.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:05.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:05.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:05.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:05.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:05.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:06.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726019 2024-09-15T04:52:06.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726019 '!=' null 2024-09-15T04:52:06.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:06.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:06.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:06.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:06.669 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:06.670 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:06.671 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:06.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:06.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:06.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:06.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:06.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:06.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:06.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:07.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:07.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:07.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:07.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:07.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:07.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:07.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:07.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:08.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589000 2024-09-15T04:52:08.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589000 '!=' null 2024-09-15T04:52:08.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:08.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:08.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:08.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:08.268 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:08.269 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:08.270 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:08.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:08.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:08.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:08.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:08.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:08.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:08.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:08.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:08.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:08.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:08.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:08.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:08.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:09.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:09.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:09.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:09.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725980 2024-09-15T04:52:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725980 '!=' null 2024-09-15T04:52:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:09.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:09.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:09.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:10.026 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:10.026 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:10.027 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:10.028 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:10.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:10.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:10.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:10.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:10.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:10.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:10.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:10.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:10.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:10.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:10.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:10.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:10.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:10.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:10.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:10.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:10.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:11.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726120 2024-09-15T04:52:11.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726120 '!=' null 2024-09-15T04:52:11.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:11.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:11.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:11.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:11.618 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:11.619 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:11.621 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:11.622 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:11.622 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:11.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:11.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:11.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:11.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:12.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:12.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:12.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:12.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:12.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:12.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:12.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:12.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:12.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:12.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:12.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:12.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:12.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589064 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589064 '!=' null 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:13.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:13.234 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:13.236 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:13.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:13.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:13.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:13.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:13.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:13.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:13.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:13.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:14.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:14.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:14.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726058 2024-09-15T04:52:14.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726058 '!=' null 2024-09-15T04:52:14.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:14.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:14.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:14.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:14.893 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:14.894 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:14.895 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:14.896 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:14.897 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:14.897 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:14.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:15.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:15.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:15.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:15.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:15.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:15.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:15.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:15.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:15.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:15.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:15.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:15.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:15.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:16.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726058 2024-09-15T04:52:16.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726058 '!=' null 2024-09-15T04:52:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:16.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:16.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:16.410 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:16.411 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:16.412 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:16.413 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:16.413 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:16.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:16.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:16.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:16.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:16.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:16.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:16.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:16.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:16.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:16.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:16.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:16.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:16.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:17.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:17.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:17.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:18.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462719 2024-09-15T04:52:18.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462719 '!=' null 2024-09-15T04:52:18.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:18.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:18.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:18.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:18.074 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:18.075 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:18.076 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:18.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:18.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:18.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:18.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:18.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:18.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:18.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:19.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:19.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:19.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:19.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:19.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599631 2024-09-15T04:52:19.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599631 '!=' null 2024-09-15T04:52:19.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:19.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:19.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:19.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:19.685 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:19.686 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:19.687 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:19.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:19.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:19.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:19.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:20.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:20.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:20.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:20.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:20.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:20.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:20.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:20.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599342 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599342 '!=' null 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:21.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:21.256 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:21.257 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:21.258 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:21.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:21.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:21.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:21.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:21.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:21.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:21.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:22.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:22.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462132 2024-09-15T04:52:22.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462132 '!=' null 2024-09-15T04:52:22.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:22.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:22.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:22.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:22.814 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:22.815 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:22.816 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:22.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:22.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:22.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:22.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:23.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:23.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:23.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:23.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:23.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:23.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462132 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462132 '!=' null 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:24.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:24.422 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:24.422 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:24.422 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:24.423 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:24.424 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:24.424 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:24.424 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:24.424 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:24.424 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:24.425 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:24.426 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:24.427 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:24.427 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:24.427 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:24.427 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:24.427 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:24.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:24.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:24.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:24.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:24.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:25.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:25.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599141 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599141 '!=' null 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:26.116 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:26.116 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:26.117 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:26.118 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:26.119 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:26.119 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:26.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:26.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:26.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:26.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:26.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:26.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:26.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:26.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:26.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:26.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:26.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:26.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:26.579 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:27.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:27.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:27.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24461842 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24461842 '!=' null 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:27.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:27.726 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:27.727 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:27.728 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:27.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:27.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:27.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:27.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:28.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:28.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:28.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:28.179 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:28.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:28.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:28.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725433 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725433 '!=' null 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:29.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:29.339 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:29.340 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:29.341 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:29.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:29.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:29.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:29.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:29.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:29.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:29.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:29.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:30.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:30.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:31.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725504 2024-09-15T04:52:31.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725504 '!=' null 2024-09-15T04:52:31.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:31.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:31.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:31.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:31.116 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:31.117 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:31.118 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:31.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:31.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:31.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:31.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:31.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:31.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:32.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:32.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:32.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462357 2024-09-15T04:52:32.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462357 '!=' null 2024-09-15T04:52:32.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:32.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:32.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:32.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:32.684 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:32.685 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:32.686 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:32.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:32.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:32.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:33.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:33.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:33.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:33.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:33.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:33.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:33.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:33.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:33.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:33.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:33.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:33.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:33.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462357 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462357 '!=' null 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:34.298 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:34.299 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:34.300 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:34.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:34.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:34.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:34.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:34.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:34.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:34.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:34.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:35.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:35.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599564 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599564 '!=' null 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:35.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:35.890 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:35.890 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:35.890 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:35.891 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:35.892 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:35.893 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:35.893 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:35.893 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:35.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:35.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:36.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:36.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:36.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:36.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:36.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:36.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:36.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:36.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:36.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:36.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:36.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:36.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:36.858 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462566 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462566 '!=' null 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:37.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [ 2024-09-15T04:52:37.486 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 190, 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T04:52:37.487 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "1.0", 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:37.488 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:37.489 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:37.489 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:37.489 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:37.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:37.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:37.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:37.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:37.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:37.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:38.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599466 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599466 '!=' null 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:39.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:39.620 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:39.621 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:39.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:39.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:40.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:40.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:40.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:40.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:40.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:40.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:40.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:40.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:41.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794344 2024-09-15T04:52:41.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794344 '!=' null 2024-09-15T04:52:41.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:41.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:41.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:41.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:41.453 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:41.454 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:41.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:41.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:41.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:41.562 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:41.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:52:41.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:41.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:41.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:42.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:42.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:43.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23588690 2024-09-15T04:52:43.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23588690 '!=' null 2024-09-15T04:52:43.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:43.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:43.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:43.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:43.411 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:43.412 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:43.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:43.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:43.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:43.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:43.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:43.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:43.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:43.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:44.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:44.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:44.420 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:44.421 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:44.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:44.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:44.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:44.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:44.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:44.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:44.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:44.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:45.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11794541 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11794541 '!=' null 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:45.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:46.008 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:46.009 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:46.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:46.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:46.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:46.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:46.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:46.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:46.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:46.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:46.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:46.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:46.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:46.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:46.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:46.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:47.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27521014 2024-09-15T04:52:47.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27521014 '!=' null 2024-09-15T04:52:47.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:47.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:47.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:47.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:47.591 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:47.592 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:47.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:47.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:47.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:48.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:48.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:48.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:48.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:48.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:48.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:48.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:48.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:49.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22326108 2024-09-15T04:52:49.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22326108 '!=' null 2024-09-15T04:52:49.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:49.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:49.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:49.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:49.223 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:49.223 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:49.223 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:49.223 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:49.223 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:49.224 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:49.225 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:49.225 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:49.225 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:49.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:49.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:49.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:49.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:49.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:49.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:49.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:49.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:50.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:50.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:50.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20138101 2024-09-15T04:52:50.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20138101 '!=' null 2024-09-15T04:52:50.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:50.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:50.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:50.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:50.864 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:50.865 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:50.866 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:50.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:50.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:51.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:51.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:51.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:51.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:51.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:51.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20138101 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20138101 '!=' null 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:52.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:52.621 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:52.622 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:52.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:52.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:53.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:53.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:53.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:53.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:53.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:53.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:53.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:54.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083755 2024-09-15T04:52:54.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083755 '!=' null 2024-09-15T04:52:54.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:54.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:54.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:54.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:54.203 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:54.203 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:54.203 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:54.203 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:54.203 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:54.204 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:54.205 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:54.205 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:54.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:54.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:54.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:54.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:54.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:54.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:54.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:54.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:54.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:54.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:54.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:55.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:55.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:55.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19220673 2024-09-15T04:52:55.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19220673 '!=' null 2024-09-15T04:52:55.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:55.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:55.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:55.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:55.825 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:55.825 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:55.826 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:55.827 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:55.827 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:55.828 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:55.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:55.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:55.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:55.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:56.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:56.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:56.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:56.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:56.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:56.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:56.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:56.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27083196 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27083196 '!=' null 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:57.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:57.508 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:57.509 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:57.510 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:57.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:57.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:57.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:57.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:57.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:57.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:57.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:52:58.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599336 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599336 '!=' null 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:52:59.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:52:59.116 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:52:59.117 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:52:59.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:52:59.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:52:59.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:52:59.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:52:59.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:52:59.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:52:59.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:00.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:00.157 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599407 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599407 '!=' null 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:00.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:00.852 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:00.853 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:00.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:00.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:00.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:00.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:00.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:00.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:00.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:00.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:00.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:01.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:01.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:01.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:01.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:01.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:01.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:02.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599407 2024-09-15T04:53:02.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599407 '!=' null 2024-09-15T04:53:02.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:02.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:02.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:02.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:02.527 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:02.527 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:02.528 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:02.529 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:02.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:02.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:02.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:02.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:02.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:02.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:03.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:03.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:03.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:03.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336173 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336173 '!=' null 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:04.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:04.293 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:04.294 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:04.295 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:04.296 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:04.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:04.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:04.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:04.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:04.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:04.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:04.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:04.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:05.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:05.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:05.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:05.393 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599589 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599589 '!=' null 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:06.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:06.091 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:06.092 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:06.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:06.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:06.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:06.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:06.595 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:06.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:06.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:07.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:07.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:07.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462459 2024-09-15T04:53:07.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462459 '!=' null 2024-09-15T04:53:07.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:07.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:07.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:07.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:07.745 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:07.745 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:07.745 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:07.745 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:07.746 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:07.747 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:07.747 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:07.747 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:07.747 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:07.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:07.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:07.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:07.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:08.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:08.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:08.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:08.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:08.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:08.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:08.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:08.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599680 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599680 '!=' null 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:09.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:09.373 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:09.374 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:09.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:09.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:09.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:09.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:09.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:09.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:09.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:09.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:09.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:09.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:09.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:09.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:09.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:10.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:10.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:10.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599701 2024-09-15T04:53:10.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599701 '!=' null 2024-09-15T04:53:10.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:10.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:10.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:10.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:11.034 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:11.034 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:11.034 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:11.034 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:11.035 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:11.036 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:11.036 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:11.036 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:11.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:11.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:11.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:11.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:11.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:11.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:11.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:11.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:11.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:11.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599701 2024-09-15T04:53:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599701 '!=' null 2024-09-15T04:53:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:12.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:12.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:12.563 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:12.563 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:12.563 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:12.564 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:12.565 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:12.565 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:12.565 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:12.565 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:12.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:12.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:12.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:12.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:13.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:13.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:13.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:13.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:13.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:13.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:14.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25336214 2024-09-15T04:53:14.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25336214 '!=' null 2024-09-15T04:53:14.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:14.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:14.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:14.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:14.221 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:14.222 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:14.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:14.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:14.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:14.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:14.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:14.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:14.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:14.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:15.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:15.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:15.178 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:15.178 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:15.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599576 2024-09-15T04:53:15.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599576 '!=' null 2024-09-15T04:53:15.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:15.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:15.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:15.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:15.809 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:15.810 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:15.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:16.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:16.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:16.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:16.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:16.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:16.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:16.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:16.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:16.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:16.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:16.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:16.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:16.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462533 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462533 '!=' null 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:17.270 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:17.343 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:17.343 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:17.343 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:17.343 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:17.343 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:17.344 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:17.345 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:17.345 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:17.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:17.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:17.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:17.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:17.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:17.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:17.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:17.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:17.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:17.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:17.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:17.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:17.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:18.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:18.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:18.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599689 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599689 '!=' null 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:18.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:19.043 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:19.044 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:19.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:19.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:19.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:19.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:19.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:19.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:19.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:20.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599859 2024-09-15T04:53:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599859 '!=' null 2024-09-15T04:53:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:20.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:20.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:20.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:20.589 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:20.590 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:20.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:20.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:21.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:21.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:21.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:21.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:21.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:21.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:21.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:21.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:22.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599859 2024-09-15T04:53:22.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599859 '!=' null 2024-09-15T04:53:22.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:22.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:22.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:22.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:22.417 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:22.417 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:22.417 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:22.417 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:22.418 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:22.419 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:22.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:22.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:22.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:22.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:22.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:22.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:22.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:22.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:23.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:23.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:23.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463003 2024-09-15T04:53:23.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463003 '!=' null 2024-09-15T04:53:23.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:23.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:23.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:23.946 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:24.018 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:24.018 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:24.018 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:24.018 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:24.018 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:24.019 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:24.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:24.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:24.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:24.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:24.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:24.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:24.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:24.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:24.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:24.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726291 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726291 '!=' null 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:25.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:25.626 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:25.627 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:25.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:25.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:25.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:25.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:26.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:26.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:26.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:26.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:26.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:26.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:26.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:26.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:26.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:26.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:26.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:26.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:26.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:27.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463127 2024-09-15T04:53:27.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463127 '!=' null 2024-09-15T04:53:27.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:27.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:27.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:27.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:27.355 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:27.356 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:27.357 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:27.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:27.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:27.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:27.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:27.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:27.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:27.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:28.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:28.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:28.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16600013 2024-09-15T04:53:28.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16600013 '!=' null 2024-09-15T04:53:28.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:28.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:28.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:28.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:28.848 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:28.848 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:28.848 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:28.849 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:28.850 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:28.850 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:28.850 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:28.850 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:28.850 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:28.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:28.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:28.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:28.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:29.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:29.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:29.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:29.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:30.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:30.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:30.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599978 2024-09-15T04:53:30.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599978 '!=' null 2024-09-15T04:53:30.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:30.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:30.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:30.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:30.658 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:30.658 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:30.658 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:30.659 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:30.660 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:30.660 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:30.660 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:30.660 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:30.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:30.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:30.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:30.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:31.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:31.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:31.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:31.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:31.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:31.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:31.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:32.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599978 2024-09-15T04:53:32.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599978 '!=' null 2024-09-15T04:53:32.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:32.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:32.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:32.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:32.439 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:32.440 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:32.441 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:32.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:32.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:32.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:32.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:32.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:32.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:32.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:32.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:32.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:32.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:32.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:32.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:32.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:33.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:33.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24463062 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24463062 '!=' null 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:34.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:34.332 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:34.332 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:34.332 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:34.332 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:34.332 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:34.333 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:34.334 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:34.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:34.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:34.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:34.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:34.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:34.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:35.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:35.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:35.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:35.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:35.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:35.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:35.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:35.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:35.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599861 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599861 '!=' null 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:35.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:35.920 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:35.920 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:35.920 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:35.920 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:35.921 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:35.922 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:35.922 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:35.922 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:35.922 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:35.923 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:35.923 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:35.923 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:35.923 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:35.923 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:35.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:36.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:36.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:36.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:36.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:36.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:36.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:36.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:36.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:36.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:36.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:36.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:36.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:36.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:36.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:36.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:37.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462698 2024-09-15T04:53:37.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462698 '!=' null 2024-09-15T04:53:37.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:37.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:37.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:37.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:37.541 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:37.542 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:37.543 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:37.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:37.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:37.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:37.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:37.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:37.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:37.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:37.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:38.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:38.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725976 2024-09-15T04:53:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725976 '!=' null 2024-09-15T04:53:39.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:39.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:39.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:39.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:39.328 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:39.329 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:39.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:39.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:39.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:39.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:39.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:39.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:39.438 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:39.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:39.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:39.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:39.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:40.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:40.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:41.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726029 2024-09-15T04:53:41.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726029 '!=' null 2024-09-15T04:53:41.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:41.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:41.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:41.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:41.105 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:41.106 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:41.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:41.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:41.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:41.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:41.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:41.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:41.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:42.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589157 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589157 '!=' null 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:42.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:42.704 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:42.704 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:42.704 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:42.705 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:42.706 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:42.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:42.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:42.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:42.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:43.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:43.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:43.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:43.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:43.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:43.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:43.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:43.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589157 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589157 '!=' null 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:44.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:44.239 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:44.239 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:44.239 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:44.240 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:44.241 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:44.241 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:44.241 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:44.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:44.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:44.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:44.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:44.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:44.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:44.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:44.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:45.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:45.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:45.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:45.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725919 2024-09-15T04:53:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725919 '!=' null 2024-09-15T04:53:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:45.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:45.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:45.932 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:45.982 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:45.983 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:45.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:46.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:46.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:46.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:46.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:46.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:46.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:46.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:46.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:47.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462585 2024-09-15T04:53:47.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462585 '!=' null 2024-09-15T04:53:47.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:47.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:47.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:47.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:47.541 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:47.542 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:47.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:47.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:47.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:47.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:47.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:47.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:47.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:48.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:48.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599702 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599702 '!=' null 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:49.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:49.113 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:49.114 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:49.115 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:49.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:49.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:49.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:49.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:49.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:49.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:49.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:49.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:49.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:49.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:49.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:49.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:49.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:50.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:50.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:50.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:50.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:50.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:50.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:50.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:50.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:50.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:50.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599640 2024-09-15T04:53:50.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599640 '!=' null 2024-09-15T04:53:50.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:50.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:50.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:50.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:50.854 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:50.854 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:50.854 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:50.854 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:50.854 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:50.855 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:50.856 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:50.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:50.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:51.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:51.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:51.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:51.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:51.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:51.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:51.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599640 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599640 '!=' null 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:52.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:52.359 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:52.360 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:52.361 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:52.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:52.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:52.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:52.469 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:52.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:52.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:52.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:52.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:53.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462490 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462490 '!=' null 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:53.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:54.036 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:54.037 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:54.038 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:54.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:54.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:54.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:54.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:54.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:54.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:54.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:54.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:54.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:54.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:54.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:54.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:54.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:55.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599363 2024-09-15T04:53:55.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599363 '!=' null 2024-09-15T04:53:55.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:55.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:55.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:55.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:55.713 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:55.714 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:55.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:55.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:55.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:56.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:56.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:56.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:56.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:56.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:56.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:56.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:56.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:57.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462485 2024-09-15T04:53:57.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462485 '!=' null 2024-09-15T04:53:57.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:57.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:57.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:57.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:57.402 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:57.403 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:57.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:57.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:57.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:57.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:57.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:57.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:57.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:57.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:57.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:57.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:57.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:57.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:57.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:53:58.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:53:58.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:53:58.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:53:58.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:53:59.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725894 2024-09-15T04:53:59.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725894 '!=' null 2024-09-15T04:53:59.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:53:59.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:59.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:53:59.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:53:59.088 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:53:59.088 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:53:59.089 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:53:59.090 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:53:59.090 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:53:59.090 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:53:59.090 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:53:59.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:53:59.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:53:59.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:53:59.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:53:59.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:53:59.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:53:59.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:53:59.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:53:59.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:53:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:53:59.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:53:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:53:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:00.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725954 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725954 '!=' null 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:00.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:00.678 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:00.679 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:00.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:00.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:00.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:01.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:01.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:01.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:01.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:01.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:01.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:01.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:01.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:02.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725954 2024-09-15T04:54:02.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725954 '!=' null 2024-09-15T04:54:02.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:02.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:02.271 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:02.271 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:02.344 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:02.345 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:02.346 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:02.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:02.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:02.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:02.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:02.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:02.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:02.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:02.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:02.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:02.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:02.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:02.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:02.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:03.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:03.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23589024 2024-09-15T04:54:03.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23589024 '!=' null 2024-09-15T04:54:03.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:03.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:03.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:03.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:03.955 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:03.956 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:03.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:04.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:04.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:04.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:04.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:04.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:04.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:05.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726024 2024-09-15T04:54:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726024 '!=' null 2024-09-15T04:54:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:05.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:05.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:05.803 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:05.804 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:05.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:05.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:05.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:05.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:06.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:06.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:06.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:06.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:06.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:07.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15726054 2024-09-15T04:54:07.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15726054 '!=' null 2024-09-15T04:54:07.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:07.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:07.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:07.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:07.404 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:07.404 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:07.404 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:07.404 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:07.405 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:07.406 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:07.406 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:07.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:07.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:07.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:07.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:07.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:07.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:07.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:07.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:07.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:07.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:07.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:07.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:07.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:08.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:08.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599761 2024-09-15T04:54:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599761 '!=' null 2024-09-15T04:54:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:08.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:08.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:09.022 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:09.023 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:09.024 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:09.025 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:09.026 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:09.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:09.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:09.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:09.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:09.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:09.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:10.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:10.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599761 2024-09-15T04:54:10.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599761 '!=' null 2024-09-15T04:54:10.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:10.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:10.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:10.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:10.692 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:10.692 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:10.693 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:10.694 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:10.694 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:10.694 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:10.694 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:10.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:10.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:11.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:11.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:11.147 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:11.147 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:11.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:11.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:11.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:11.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:11.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:11.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:11.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:11.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:11.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:12.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599712 2024-09-15T04:54:12.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599712 '!=' null 2024-09-15T04:54:12.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:12.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:12.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:12.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:12.226 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:12.227 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:12.228 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:12.228 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:12.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:12.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:12.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:12.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:12.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:12.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:12.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:12.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:12.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:12.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:12.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:12.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:12.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:13.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:13.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:13.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:13.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:13.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462711 2024-09-15T04:54:13.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462711 '!=' null 2024-09-15T04:54:13.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:13.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:13.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:13.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:13.966 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:13.967 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:13.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:14.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:14.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:14.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:14.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:14.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:15.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:15.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:15.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:15.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599505 2024-09-15T04:54:15.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599505 '!=' null 2024-09-15T04:54:15.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:15.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:15.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:15.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:15.781 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:15.781 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:15.781 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:15.782 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:15.783 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:15.783 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:15.783 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:15.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:15.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:15.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:15.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:15.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:15.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:15.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:15.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:15.891 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:16.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:16.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:16.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:16.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:16.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:16.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16599366 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16599366 '!=' null 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:17.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:17.585 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:17.585 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:17.585 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:17.586 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:17.587 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:17.587 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:17.587 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:17.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:17.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:17.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:17.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:17.695 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:17.695 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:18.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:18.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:18.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:18.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:18.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462017 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462017 '!=' null 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:19.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:19.270 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:19.271 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:19.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:19.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:19.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:19.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:19.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:19.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:19.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:19.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:20.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:20.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:20.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725566 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725566 '!=' null 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:20.836 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:20.908 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:20.909 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:20.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:21.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:21.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:21.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:21.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:21.364 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:21.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:22.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725566 2024-09-15T04:54:22.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725566 '!=' null 2024-09-15T04:54:22.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:22.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:22.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:22.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:22.664 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:22.664 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:22.664 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:22.664 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:22.665 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:22.665 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [ 2024-09-15T04:54:22.665 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:22.665 INFO:tasks.workunit.client.0.smithi138.stdout: "item": "3.0", 2024-09-15T04:54:22.665 INFO:tasks.workunit.client.0.smithi138.stdout: "prio": 190, 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "can_preempt": true 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:22.668 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:22.669 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:22.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:22.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:22.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:22.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:22.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:22.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:22.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:22.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:22.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:23.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:23.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:23.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:23.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:23.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:24.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24462032 2024-09-15T04:54:24.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24462032 '!=' null 2024-09-15T04:54:24.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:24.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:24.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:24.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:24.290 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:24.291 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:24.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:24.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:24.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:24.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:24.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:24.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:25.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:25.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:25.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:25.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19919111 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19919111 '!=' null 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:25.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:25.956 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:25.957 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:25.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:26.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:26.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:26.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:26.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:26.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:26.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:26.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:26.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:26.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:27.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13104772 2024-09-15T04:54:27.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13104772 '!=' null 2024-09-15T04:54:27.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:27.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:27.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:27.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:27.431 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:27.432 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:27.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:27.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:27.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:27.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:27.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T04:54:27.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:27.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:27.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:28.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-09-15T04:54:28.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-09-15T04:54:28.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T04:54:28.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T04:54:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19657164 2024-09-15T04:54:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19657164 '!=' null 2024-09-15T04:54:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T04:54:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:28.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-09-15T04:54:28.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_recovery_reservations 2024-09-15T04:54:28.925 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:28.925 INFO:tasks.workunit.client.0.smithi138.stdout: "local_reservations": { 2024-09-15T04:54:28.925 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:28.925 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:28.925 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "remote_reservations": { 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "max_allowed": 1, 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "min_priority": 0, 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "queues": [], 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: "in_progress": [] 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:28.926 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:28.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:54:29.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:54:29.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:54:29.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:54:29.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-09-15T04:54:29.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:54:29.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:54:29.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:54:29.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-09-15T04:54:29.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:54:29.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:54:29.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-09-15T04:54:30.174 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:54:30.174 INFO:tasks.workunit.client.0.smithi138.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:50:53.635047+0000 95'600 122:1104 [1,4] 1 [1,4] 1 0'0 2024-09-15T04:42:54.403964+0000 0'0 2024-09-15T04:42:54.403964+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:32:11.955113+0000 0 0 2024-09-15T04:54:30.174 INFO:tasks.workunit.client.0.smithi138.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:54:24.960181+0000 95'600 123:1115 [1,2] 1 [1,2] 1 0'0 2024-09-15T04:42:42.819245+0000 0'0 2024-09-15T04:42:42.819245+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:48:33.461384+0000 0 0 2024-09-15T04:54:30.174 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-09-15T04:52:40.314819+0000 95'600 122:1107 [1,2] 1 [1,2] 1 0'0 2024-09-15T04:42:37.950625+0000 0'0 2024-09-15T04:42:37.950625+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:48:47.409803+0000 0 0 2024-09-15T04:54:30.175 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:54:30.175 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:54:30.175 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:54:30.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.215650/ceph-osd.1.asok 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-09-15T04:54:30.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.215650/ceph-osd.1.asok dump_pgstate_history 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "pgs": [ 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "pg": "7.0", 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "history": [ 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "64", 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.260 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.101875+0000", 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.101882+0000" 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.101886+0000", 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.261432+0000" 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.261 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.261432+0000", 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.261483+0000" 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.261484+0000", 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.261494+0000" 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.261494+0000", 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.429312+0000" 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.262 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.101884+0000", 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.429317+0000" 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.263 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.429500+0000", 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.790980+0000" 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.790981+0000", 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:55.791033+0000" 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.791034+0000", 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369396+0000" 2024-09-15T04:54:30.264 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.429318+0000", 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369418+0000" 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.101882+0000", 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369438+0000" 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.265 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:55.101873+0000", 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369441+0000" 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "64", 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369442+0000", 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369730+0000" 2024-09-15T04:54:30.266 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "93", 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369735+0000", 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.369745+0000" 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.267 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369751+0000", 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.488117+0000" 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.488117+0000", 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.488212+0000" 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.488212+0000", 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.488224+0000" 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.268 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.488224+0000", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.927820+0000" 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369748+0000", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:56.927833+0000" 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.928021+0000", 2024-09-15T04:54:30.269 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:57.192588+0000" 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:57.192589+0000", 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:57.192649+0000" 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:57.192649+0000", 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.333758+0000" 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.270 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.927834+0000", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.333777+0000" 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369745+0000", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.333793+0000" 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:56.369731+0000", 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.333796+0000" 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.271 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "93", 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.333799+0000", 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.334088+0000" 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.272 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "100", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334092+0000", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.334100+0000" 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334106+0000", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.334232+0000" 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.273 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334233+0000", 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.334278+0000" 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334278+0000", 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.334289+0000" 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334289+0000", 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.794450+0000" 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.274 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334103+0000", 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.794456+0000" 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.794509+0000", 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.871898+0000" 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.275 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.871899+0000", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:24.871924+0000" 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.871924+0000", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058041+0000" 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.794457+0000", 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058062+0000" 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.276 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334101+0000", 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058193+0000" 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:24.334088+0000", 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058197+0000" 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.277 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "101", 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058197+0000", 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058627+0000" 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "103", 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.278 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058630+0000", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.058641+0000" 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058648+0000", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.184052+0000" 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.184052+0000", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.184110+0000" 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.279 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058644+0000", 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.184117+0000" 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.184117+0000", 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.662744+0000" 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058641+0000", 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.662772+0000" 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.280 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.058627+0000", 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.662775+0000" 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "103", 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.281 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.662776+0000", 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.663131+0000" 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "118", 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663136+0000", 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.663148+0000" 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.282 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663155+0000", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.663662+0000" 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663663+0000", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.663722+0000" 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663722+0000", 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:17.663734+0000" 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.283 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663735+0000", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:18.175760+0000" 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663151+0000", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:18.175768+0000" 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:18.177287+0000", 2024-09-15T04:54:30.284 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:18.446031+0000" 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:18.446032+0000", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:18.446186+0000" 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:18.446187+0000", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:18.446503+0000" 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:18.446504+0000", 2024-09-15T04:54:30.285 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:49.368689+0000" 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:49.368690+0000", 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:49.368919+0000" 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:49.368920+0000", 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:49:01.344534+0000" 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-15T04:54:30.286 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:49:01.344534+0000", 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:49:01.344836+0000" 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:49:01.344836+0000", 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.088554+0000" 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:50:52.088555+0000", 2024-09-15T04:54:30.287 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.481695+0000" 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:18.175769+0000", 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.481733+0000" 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663148+0000", 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.481770+0000" 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.288 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:17.663132+0000", 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.481821+0000" 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "118", 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:50:52.481822+0000", 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.482490+0000" 2024-09-15T04:54:30.289 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "pg": "3.0", 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "history": [ 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "40", 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.290 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.754025+0000", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:43.754033+0000" 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.754039+0000", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:43.871804+0000" 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.871804+0000", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:43.871884+0000" 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.871885+0000", 2024-09-15T04:54:30.291 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:43.871899+0000" 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.871900+0000", 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:44.131832+0000" 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.754036+0000", 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:44.131837+0000" 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.292 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:44.132015+0000", 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073039+0000" 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:44.131846+0000", 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073053+0000" 2024-09-15T04:54:30.293 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.754034+0000", 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073069+0000" 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:43.754022+0000", 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073074+0000" 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.294 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "40", 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073075+0000", 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073347+0000" 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "90", 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.295 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073349+0000", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.073357+0000" 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073362+0000", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.208048+0000" 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.208048+0000", 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.208099+0000" 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.296 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.208100+0000", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.208109+0000" 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.208109+0000", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.484495+0000" 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073359+0000", 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.484504+0000" 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.297 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.484683+0000", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.751140+0000" 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.751140+0000", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:45.751175+0000" 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.751175+0000", 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.520691+0000" 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.298 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.484509+0000", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.520720+0000" 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073357+0000", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.520751+0000" 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:45.073347+0000", 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.520757+0000" 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.299 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "90", 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.520757+0000", 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.521209+0000" 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.300 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "109", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521215+0000", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.521230+0000" 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521240+0000", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.521638+0000" 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.301 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521638+0000", 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.521721+0000" 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521722+0000", 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.521743+0000" 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521743+0000", 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.897915+0000" 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.302 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521235+0000", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.897922+0000" 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.897977+0000", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.991129+0000" 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.991129+0000", 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:43:23.991197+0000" 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.303 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.991197+0000", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.030614+0000" 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.897923+0000", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.030640+0000" 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521230+0000", 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.030886+0000" 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.304 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:43:23.521209+0000", 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.030889+0000" 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "110", 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.030890+0000", 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.031315+0000" 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.305 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "112", 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.031320+0000", 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.031331+0000" 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.031338+0000", 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.148214+0000" 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.306 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.148215+0000", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.148262+0000" 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.031335+0000", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.148269+0000" 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.148269+0000", 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.778125+0000" 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.307 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.031332+0000", 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.778186+0000" 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.031315+0000", 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.778189+0000" 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.308 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "112", 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778190+0000", 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.778527+0000" 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "122", 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.309 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778531+0000", 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.778543+0000" 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778550+0000", 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.779178+0000" 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.779179+0000", 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.779276+0000" 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.310 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.779276+0000", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:36.779300+0000" 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.779301+0000", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:37.223861+0000" 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778546+0000", 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:37.223880+0000" 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.311 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:37.225401+0000", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:37.392233+0000" 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:37.392234+0000", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:49.368848+0000" 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:49.368849+0000", 2024-09-15T04:54:30.312 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:49.369157+0000" 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:49.369162+0000", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:49:01.344398+0000" 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:49:01.344399+0000", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:49:01.344578+0000" 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:49:01.344579+0000", 2024-09-15T04:54:30.313 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.231654+0000" 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:52:38.231655+0000", 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.231959+0000" 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:52:38.231960+0000", 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.019875+0000" 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.314 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:54:24.019876+0000", 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.313540+0000" 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:37.223881+0000", 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.313559+0000" 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778543+0000", 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.313584+0000" 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.315 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:36.778527+0000", 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.313588+0000" 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "122", 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:54:24.313589+0000", 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:54:24.314028+0000" 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.316 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "pg": "1.0", 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "currently": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "history": [ 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "27", 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Initial", 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513638+0000", 2024-09-15T04:54:30.317 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.513778+0000" 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "27", 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513778+0000", 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.513806+0000" 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.318 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "29", 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513815+0000", 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.513824+0000" 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513830+0000", 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.514288+0000" 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.319 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.514289+0000", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.514339+0000" 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.514340+0000", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.514348+0000" 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.514349+0000", 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.891333+0000" 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.320 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513827+0000", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:38.891341+0000" 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.891491+0000", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.072718+0000" 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.891342+0000", 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.072735+0000" 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.321 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513824+0000", 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.072755+0000" 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:38.513807+0000", 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.072758+0000" 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "30", 2024-09-15T04:54:30.322 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.072759+0000", 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.073046+0000" 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "105", 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073049+0000", 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.073056+0000" 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.323 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073061+0000", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.073599+0000" 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073600+0000", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.073654+0000" 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073654+0000", 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.073663+0000" 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.324 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073663+0000", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.416889+0000" 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073059+0000", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.416903+0000" 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.417094+0000", 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.941375+0000" 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.325 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.941375+0000", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:42:40.941410+0000" 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Clean", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.941411+0000", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917252+0000" 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.416904+0000", 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917274+0000" 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.326 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073056+0000", 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917411+0000" 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:42:40.073046+0000", 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917415+0000" 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "105", 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.327 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917418+0000", 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917826+0000" 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "107", 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.328 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917830+0000", 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:28.917840+0000" 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917847+0000", 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.018175+0000" 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.018176+0000", 2024-09-15T04:54:30.329 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.018237+0000" 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917844+0000", 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.018244+0000" 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/WaitActingChange", 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.018244+0000", 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.519428+0000" 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.330 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917841+0000", 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.519455+0000" 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:28.917826+0000", 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.519459+0000" 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "107", 2024-09-15T04:54:30.331 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519460+0000", 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.519774+0000" 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "120", 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Start", 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519779+0000", 2024-09-15T04:54:30.332 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.519790+0000" 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519797+0000", 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.664908+0000" 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetLog", 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.664909+0000", 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.664980+0000" 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.333 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.664980+0000", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.664993+0000" 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.664994+0000", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.975446+0000" 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Peering", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519794+0000", 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:29.975454+0000" 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.334 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Activating", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.977086+0000", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:48:30.412129+0000" 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:30.412130+0000", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.088682+0000" 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:50:52.088683+0000", 2024-09-15T04:54:30.335 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:50:52.089025+0000" 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovering", 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:50:52.089030+0000", 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.231455+0000" 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active/Recovered", 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:52:38.231456+0000", 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.615434+0000" 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary/Active", 2024-09-15T04:54:30.336 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.975455+0000", 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.615459+0000" 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started/Primary", 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519791+0000", 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.615483+0000" 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Started", 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:48:29.519775+0000", 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.615486+0000" 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.337 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": "120", 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: "states": [ 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "Reset", 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: "enter": "2024-09-15T04:52:38.615487+0000", 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: "exit": "2024-09-15T04:52:38.615906+0000" 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T04:54:30.338 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T04:54:30.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-09-15T04:54:30.339 INFO:tasks.workunit.client.0.smithi138.stdout:TEST PASSED 2024-09-15T04:54:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-09-15T04:54:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-09-15T04:54:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-15T04:54:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-15T04:54:30.810 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' does not exist 2024-09-15T04:54:30.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-09-15T04:54:30.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-09-15T04:54:30.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-09-15T04:54:31.414 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test3' does not exist 2024-09-15T04:54:31.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-09-15T04:54:31.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-09-15T04:54:31.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-09-15T04:54:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test7' does not exist 2024-09-15T04:54:32.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-09-15T04:54:32.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:54:32.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:54:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:54:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:54:32.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:54:37.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:54:37.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:54:37.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:54:37.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:54:37.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:54:37.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:54:37.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:54:37.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:54:37.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:54:37.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:54:37.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:54:37.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:54:37.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:54:37.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:54:37.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-15T04:54:37.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:54:37.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:37.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:54:37.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215650 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-prio 0 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:54:37.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:54:37.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:54:37.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:54:37.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:54:37.532 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:54:37.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:54:37.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:54:37.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:54:37.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:54:37.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:54:37.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:54:37.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:54:37.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:54:37.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:54:37.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-09-15T04:54:37.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:54:37.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:37.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.215650 2024-09-15T04:54:37.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.215650 2024-09-15T04:54:37.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:54:37.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:54:37.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:54:37.541 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:54:37.542 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:54:37.567 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-09-15T04:54:37.567 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-09-15T04:54:37.632 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-space 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-09-15T04:54:37.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-09-15T04:54:37.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-09-15T04:54:37.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-09-15T04:54:37.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-09-15T04:54:37.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none ' 2024-09-15T04:54:37.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-09-15T04:54:37.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-09-15T04:54:37.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:54:37.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:54:37.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:54:37.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:54:37.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:54:37.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:54:37.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:54:37.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:54:37.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:54:37.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:54:37.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:54:37.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:54:37.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:54:37.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:54:37.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-15T04:54:37.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:54:37.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:37.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:37.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.273887 2024-09-15T04:54:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:54:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:54:37.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-09-15T04:54:37.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:54:37.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:37.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:37.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.273887 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: local pools=1 2024-09-15T04:54:37.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: local OSDS=2 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-09-15T04:54:37.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:54:38.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:54:38.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:54:38.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.273887/ceph-mon.a.asok 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:54:38.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.273887/ceph-mon.a.asok config get fsid 2024-09-15T04:54:38.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:54:38.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:54:38.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.273887/ceph-mon.a.asok 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:54:38.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.273887/ceph-mon.a.asok config get mon_host 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:98: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-09-15T04:54:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:54:39.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:54:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:54:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: export CEPH_ARGS 2024-09-15T04:54:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: expr 2 - 1 2024-09-15T04:54:39.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: seq 0 1 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-09-15T04:54:39.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:54:39.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-09-15T04:54:39.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:54:39.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135 2024-09-15T04:54:39.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135' 2024-09-15T04:54:39.285 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135 2024-09-15T04:54:39.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:54:39.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPaOZmfuW1ERAAupdmETi7nzGm5Bplzlk7+A== 2024-09-15T04:54:39.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPaOZmfuW1ERAAupdmETi7nzGm5Bplzlk7+A=="}' 2024-09-15T04:54:39.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135 -i td/osd-recovery-space/0/new.json 2024-09-15T04:54:39.595 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:54:39.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-09-15T04:54:39.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPaOZmfuW1ERAAupdmETi7nzGm5Bplzlk7+A== --osd-uuid 47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135 2024-09-15T04:54:39.631 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.630+0000 7f37e3b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:39.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.641+0000 7f37e3b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:39.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.642+0000 7f37e3b1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:39.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.642+0000 7f37e3b1d440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-15T04:54:39.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.642+0000 7f37e3b1d440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-15T04:54:39.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.642+0000 7f37e3b1d440 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-09-15T04:54:39.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:39.681+0000 7f37e3b1d440 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-09-15T04:54:45.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-09-15T04:54:45.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:54:45.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:54:45.281 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:54:45.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:54:45.771 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:54:45.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:54:45.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:54:45.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:54:45.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:54:45.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:54:45.796 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:45.795+0000 7f4f37a13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:45.798 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:45.797+0000 7f4f37a13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:45.799 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:45.798+0000 7f4f37a13440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:54:46.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:54:46.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:54:46.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:54:46.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:54:46.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:54:46.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:46.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:54:46.122 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:54:46.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:46.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:46.321 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:46.321+0000 7f4f37a13440 -1 Falling back to public interface 2024-09-15T04:54:46.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:47.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:47.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:47.501 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:54:47.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:54:47.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:47.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:47.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:48.822 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:54:48.821+0000 7f4f37a13440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:54:48.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:48.854 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:54:48.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:48.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:54:48.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:48.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:49.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:50.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:50.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:50.215 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:54:50.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:54:50.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:50.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:50.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:51.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:51.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:51.699 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:54:51.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:54:51.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:51.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:52.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:53.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:53.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:53.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:54:53.069 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:54:53.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:53.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:53.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:54.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:54.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:54.519 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:54:54.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:54:54.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:54.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:54.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:55.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:55.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:55.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:54:55.881 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:54:55.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:55.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:56.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:57.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:57.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:57.263 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:54:57.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:54:57.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:57.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:57.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:54:58.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:54:58.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:54:58.679 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:54:58.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:54:58.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:54:58.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:54:59.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:00.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:00.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:00.052 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:55:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:55:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:00.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:01.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:01.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:01.537 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:55:01.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:55:01.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:01.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:02.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:03.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:03.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:03.008 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:55:03.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:55:03.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:03.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:03.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:04.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:04.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:04.730 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:55:04.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:55:04.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:04.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:05.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:06.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:06.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:06.247 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:55:06.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:55:06.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:06.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:06.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:07.985 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:55:07.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:55:07.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:07.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:08.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:09.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:09.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:09.856 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:55:09.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:55:09.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:09.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:10.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:11.687 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:55:11.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:55:11.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:11.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:55:12.159 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/274980,v1:127.0.0.1:6803/274980] [v2:127.0.0.1:6804/274980,v1:127.0.0.1:6805/274980] exists,up 47c6e7f5-57c1-4e24-a3e8-6c2ef1a6e135 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:101: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-09-15T04:55:12.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:55:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:55:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c99c257d-875e-493b-bb84-970adf9e69e5 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c99c257d-875e-493b-bb84-970adf9e69e5' 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 c99c257d-875e-493b-bb84-970adf9e69e5 2024-09-15T04:55:12.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:55:12.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCwaOZm/0iLChAA5O/1CuO7j+WjSzJu5VFJEg== 2024-09-15T04:55:12.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCwaOZm/0iLChAA5O/1CuO7j+WjSzJu5VFJEg=="}' 2024-09-15T04:55:12.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c99c257d-875e-493b-bb84-970adf9e69e5 -i td/osd-recovery-space/1/new.json 2024-09-15T04:55:12.688 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:55:12.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-09-15T04:55:12.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCwaOZm/0iLChAA5O/1CuO7j+WjSzJu5VFJEg== --osd-uuid c99c257d-875e-493b-bb84-970adf9e69e5 2024-09-15T04:55:12.726 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.725+0000 7f131a61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:12.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.727+0000 7f131a61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:12.728 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.728+0000 7f131a61d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.728+0000 7f131a61d440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-15T04:55:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.728+0000 7f131a61d440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-15T04:55:12.729 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.728+0000 7f131a61d440 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-09-15T04:55:12.778 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:12.778+0000 7f131a61d440 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-09-15T04:55:18.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-09-15T04:55:18.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:55:18.730 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:55:18.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:55:18.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:55:19.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:55:19.211 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:55:19.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=c2a70de4-d034-48c4-a126-d95d1494073f --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.273887/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:55:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:55:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:55:19.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:55:19.235 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:19.233+0000 7f61c040a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:19.236 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:19.235+0000 7f61c040a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:19.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:19.236+0000 7f61c040a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:55:19.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:55:19.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:55:19.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:55:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:55:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:55:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:19.563 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:55:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:55:19.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:19.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:19.752 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:19.751+0000 7f61c040a440 -1 Falling back to public interface 2024-09-15T04:55:19.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:20.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:20.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:20.956 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:55:20.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:55:20.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:20.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:21.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:22.270 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:55:22.269+0000 7f61c040a440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:55:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:55:22.315 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:55:22.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:22.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:22.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:23.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:23.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:23.819 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:55:23.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:55:23.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:23.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:24.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:25.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:25.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:25.336 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:55:25.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:55:25.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:25.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:25.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:26.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:26.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:26.735 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:55:26.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:55:26.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:26.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:27.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:28.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:28.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:28.251 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:55:28.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:55:28.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:28.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:28.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:29.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:29.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:29.622 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:55:29.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:55:29.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:29.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:30.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:31.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:31.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:31.025 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:55:31.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:55:31.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:31.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:31.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:32.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:32.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:32.396 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:55:32.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:55:32.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:32.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:32.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:33.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:33.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:33.778 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:55:33.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:55:33.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:33.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:34.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:35.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:35.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:35.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:55:35.252 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:55:35.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:35.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:36.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:37.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:37.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:37.073 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:55:37.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:55:37.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:37.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:39.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:39.271 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:55:39.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:55:39.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:39.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:40.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:41.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:41.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:41.025 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:55:41.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:55:41.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:41.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:41.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:42.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:42.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:42.848 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:55:42.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:55:42.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:42.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:44.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:44.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:44.795 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:55:44.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:55:44.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:44.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:45.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:55:46.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:55:46.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:55:46.904 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:55:46.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:55:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:55:46.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:55:47.260 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/276734,v1:127.0.0.1:6811/276734] [v2:127.0.0.1:6812/276734,v1:127.0.0.1:6813/276734] exists,up c99c257d-875e-493b-bb84-970adf9e69e5 2024-09-15T04:55:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:55:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:55:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:55:47.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-09-15T04:55:47.947 INFO:tasks.workunit.client.0.smithi138.stderr:osd set-nearfull-ratio 0.4 2024-09-15T04:55:47.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:107: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-09-15T04:55:48.776 INFO:tasks.workunit.client.0.smithi138.stderr:osd set-backfillfull-ratio 0.45 2024-09-15T04:55:48.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-09-15T04:55:49.604 INFO:tasks.workunit.client.0.smithi138.stderr:osd set-full-ratio 0.5 2024-09-15T04:55:49.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:110: TEST_recovery_test_simple: seq 1 1 2024-09-15T04:55:49.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:110: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-15T04:55:49.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: create_pool test1 1 1 2024-09-15T04:55:49.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-09-15T04:55:50.443 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' already exists 2024-09-15T04:55:50.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T04:55:51.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:113: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-09-15T04:55:52.168 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 1 2024-09-15T04:55:52.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:116: TEST_recovery_test_simple: wait_for_clean 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:55:52.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:55:52.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:55:52.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:55:52.747 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T04:55:52.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:55:52.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:55:52.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:55:52.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836497 2024-09-15T04:55:52.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836497 2024-09-15T04:55:52.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836497' 2024-09-15T04:55:52.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:55:52.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:55:53.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672968 2024-09-15T04:55:53.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672968 2024-09-15T04:55:53.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836497 1-42949672968' 2024-09-15T04:55:53.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:55:53.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836497 2024-09-15T04:55:53.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:55:53.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:55:53.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836497 2024-09-15T04:55:53.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:55:53.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836497 2024-09-15T04:55:53.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2024-09-15T04:55:53.016 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836497 2024-09-15T04:55:53.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:55:53.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836497 2024-09-15T04:55:53.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:55:54.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:55:54.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:55:54.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836495 -lt 21474836497 2024-09-15T04:55:54.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:55:55.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T04:55:55.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:55:56.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836497 -lt 21474836497 2024-09-15T04:55:56.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:55:56.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672968 2024-09-15T04:55:56.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:55:56.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:55:56.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672968 2024-09-15T04:55:56.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:55:56.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672968 2024-09-15T04:55:56.083 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672968 2024-09-15T04:55:56.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-09-15T04:55:56.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:55:56.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672968 2024-09-15T04:55:56.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T04:55:56.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:55:56.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:55:56.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T04:55:56.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T04:55:56.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T04:55:57.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T04:55:57.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T04:55:57.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:55:57.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:55:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T04:55:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T04:55:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T04:55:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:118: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-09-15T04:55:57.779 INFO:tasks.workunit.client.0.smithi138.stderr:5+0 records in 2024-09-15T04:55:57.780 INFO:tasks.workunit.client.0.smithi138.stderr:5+0 records out 2024-09-15T04:55:57.780 INFO:tasks.workunit.client.0.smithi138.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 6.3781e-05 s, 80.3 MB/s 2024-09-15T04:55:57.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: seq 1 600 2024-09-15T04:55:57.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:57.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-09-15T04:55:57.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:57.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-09-15T04:55:58.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-09-15T04:55:58.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-09-15T04:55:58.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-09-15T04:55:58.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-09-15T04:55:58.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-09-15T04:55:58.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-09-15T04:55:58.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-09-15T04:55:58.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-09-15T04:55:58.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-09-15T04:55:58.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-09-15T04:55:58.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-09-15T04:55:58.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:58.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-09-15T04:55:59.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-09-15T04:55:59.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-09-15T04:55:59.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-09-15T04:55:59.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-09-15T04:55:59.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-09-15T04:55:59.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-09-15T04:55:59.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-09-15T04:55:59.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-09-15T04:55:59.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-09-15T04:55:59.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-09-15T04:55:59.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-09-15T04:55:59.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:55:59.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-09-15T04:56:00.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-09-15T04:56:00.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-09-15T04:56:00.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-09-15T04:56:00.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-09-15T04:56:00.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-09-15T04:56:00.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-09-15T04:56:00.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-09-15T04:56:00.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-09-15T04:56:00.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-09-15T04:56:00.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-09-15T04:56:00.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-09-15T04:56:00.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:00.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-09-15T04:56:01.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-09-15T04:56:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-09-15T04:56:01.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-09-15T04:56:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-09-15T04:56:01.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-09-15T04:56:01.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-09-15T04:56:01.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-09-15T04:56:01.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-09-15T04:56:01.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-09-15T04:56:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-09-15T04:56:01.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-09-15T04:56:01.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-09-15T04:56:01.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:01.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-09-15T04:56:02.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-09-15T04:56:02.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-09-15T04:56:02.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-09-15T04:56:02.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-09-15T04:56:02.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-09-15T04:56:02.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-09-15T04:56:02.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-09-15T04:56:02.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-09-15T04:56:02.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-09-15T04:56:02.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-09-15T04:56:02.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-09-15T04:56:02.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:02.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-09-15T04:56:03.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-09-15T04:56:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-09-15T04:56:03.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-09-15T04:56:03.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-09-15T04:56:03.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-09-15T04:56:03.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-09-15T04:56:03.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-09-15T04:56:03.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-09-15T04:56:03.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-09-15T04:56:03.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-09-15T04:56:03.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-09-15T04:56:03.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:03.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-09-15T04:56:04.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-09-15T04:56:04.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-09-15T04:56:04.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-09-15T04:56:04.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-09-15T04:56:04.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-09-15T04:56:04.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-09-15T04:56:04.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-09-15T04:56:04.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-09-15T04:56:04.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-09-15T04:56:04.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-09-15T04:56:04.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-09-15T04:56:04.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:04.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-09-15T04:56:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-09-15T04:56:05.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-09-15T04:56:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-09-15T04:56:05.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-09-15T04:56:05.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-09-15T04:56:05.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-09-15T04:56:05.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-09-15T04:56:05.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-09-15T04:56:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-09-15T04:56:05.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-09-15T04:56:05.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-09-15T04:56:05.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:05.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-09-15T04:56:06.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-09-15T04:56:06.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-09-15T04:56:06.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-09-15T04:56:06.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-09-15T04:56:06.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-09-15T04:56:06.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-09-15T04:56:06.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-09-15T04:56:06.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-09-15T04:56:06.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-09-15T04:56:06.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-09-15T04:56:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-09-15T04:56:06.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-09-15T04:56:06.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:06.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-09-15T04:56:07.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-09-15T04:56:07.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-09-15T04:56:07.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-09-15T04:56:07.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-09-15T04:56:07.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-09-15T04:56:07.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-09-15T04:56:07.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-09-15T04:56:07.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-09-15T04:56:07.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-09-15T04:56:07.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-09-15T04:56:07.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-09-15T04:56:07.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:07.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-09-15T04:56:08.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-09-15T04:56:08.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-09-15T04:56:08.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-09-15T04:56:08.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-09-15T04:56:08.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-09-15T04:56:08.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-09-15T04:56:08.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-09-15T04:56:08.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-09-15T04:56:08.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-09-15T04:56:08.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-09-15T04:56:08.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-09-15T04:56:08.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:08.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-09-15T04:56:09.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-09-15T04:56:09.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-09-15T04:56:09.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-09-15T04:56:09.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-09-15T04:56:09.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-09-15T04:56:09.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-09-15T04:56:09.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-09-15T04:56:09.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-09-15T04:56:09.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-09-15T04:56:09.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-09-15T04:56:09.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-09-15T04:56:09.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:09.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-09-15T04:56:10.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-09-15T04:56:10.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-09-15T04:56:10.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-09-15T04:56:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-09-15T04:56:10.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-09-15T04:56:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-09-15T04:56:10.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-09-15T04:56:10.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-09-15T04:56:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-09-15T04:56:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-09-15T04:56:10.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-09-15T04:56:10.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-09-15T04:56:10.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:10.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-09-15T04:56:11.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-09-15T04:56:11.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-09-15T04:56:11.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-09-15T04:56:11.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-09-15T04:56:11.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-09-15T04:56:11.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-09-15T04:56:11.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-09-15T04:56:11.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-09-15T04:56:11.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-09-15T04:56:11.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-09-15T04:56:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-09-15T04:56:11.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:11.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-09-15T04:56:12.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-09-15T04:56:12.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-09-15T04:56:12.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-09-15T04:56:12.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-09-15T04:56:12.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-09-15T04:56:12.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-09-15T04:56:12.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-09-15T04:56:12.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-09-15T04:56:12.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-09-15T04:56:12.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-09-15T04:56:12.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-09-15T04:56:12.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:12.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-09-15T04:56:13.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-09-15T04:56:13.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-09-15T04:56:13.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-09-15T04:56:13.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-09-15T04:56:13.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-09-15T04:56:13.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-09-15T04:56:13.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-09-15T04:56:13.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:13.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-09-15T04:56:14.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-09-15T04:56:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-09-15T04:56:14.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-09-15T04:56:14.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-09-15T04:56:14.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-09-15T04:56:14.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-09-15T04:56:14.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-09-15T04:56:14.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-09-15T04:56:14.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-09-15T04:56:14.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-09-15T04:56:14.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-09-15T04:56:14.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-09-15T04:56:14.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:14.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-09-15T04:56:15.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-09-15T04:56:15.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-09-15T04:56:15.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-09-15T04:56:15.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-09-15T04:56:15.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-09-15T04:56:15.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-09-15T04:56:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-09-15T04:56:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-09-15T04:56:15.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-09-15T04:56:15.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-09-15T04:56:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-09-15T04:56:15.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:15.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-09-15T04:56:16.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-09-15T04:56:16.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-09-15T04:56:16.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-09-15T04:56:16.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-09-15T04:56:16.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-09-15T04:56:16.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-09-15T04:56:16.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-09-15T04:56:16.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-09-15T04:56:16.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-09-15T04:56:16.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-09-15T04:56:16.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-09-15T04:56:16.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-09-15T04:56:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-09-15T04:56:17.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-09-15T04:56:17.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-09-15T04:56:17.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-09-15T04:56:17.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-09-15T04:56:17.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-09-15T04:56:17.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-09-15T04:56:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-09-15T04:56:17.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-09-15T04:56:17.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-09-15T04:56:17.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-09-15T04:56:17.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-09-15T04:56:17.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:17.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-09-15T04:56:18.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-09-15T04:56:18.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-09-15T04:56:18.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-09-15T04:56:18.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-09-15T04:56:18.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-09-15T04:56:18.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-09-15T04:56:18.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-09-15T04:56:18.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-09-15T04:56:18.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-09-15T04:56:18.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-09-15T04:56:18.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:18.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-09-15T04:56:19.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-09-15T04:56:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-09-15T04:56:19.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-09-15T04:56:19.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-09-15T04:56:19.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-09-15T04:56:19.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-09-15T04:56:19.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-09-15T04:56:19.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-09-15T04:56:19.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-09-15T04:56:19.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-09-15T04:56:19.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-09-15T04:56:19.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:19.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-09-15T04:56:20.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-09-15T04:56:20.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-09-15T04:56:20.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-09-15T04:56:20.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-09-15T04:56:20.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-09-15T04:56:20.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-09-15T04:56:20.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-09-15T04:56:20.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-09-15T04:56:20.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-09-15T04:56:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-09-15T04:56:20.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-09-15T04:56:20.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-09-15T04:56:20.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:20.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-09-15T04:56:21.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-09-15T04:56:21.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-09-15T04:56:21.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-09-15T04:56:21.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-09-15T04:56:21.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-09-15T04:56:21.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-09-15T04:56:21.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-09-15T04:56:21.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-09-15T04:56:21.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-09-15T04:56:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-09-15T04:56:21.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-09-15T04:56:21.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:21.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-09-15T04:56:22.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-09-15T04:56:22.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-09-15T04:56:22.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-09-15T04:56:22.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-09-15T04:56:22.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-09-15T04:56:22.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-09-15T04:56:22.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-09-15T04:56:22.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-09-15T04:56:22.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-09-15T04:56:22.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-09-15T04:56:22.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-09-15T04:56:22.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:22.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-09-15T04:56:23.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-09-15T04:56:23.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-09-15T04:56:23.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-09-15T04:56:23.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-09-15T04:56:23.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-09-15T04:56:23.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-09-15T04:56:23.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-09-15T04:56:23.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-09-15T04:56:23.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-09-15T04:56:23.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-09-15T04:56:23.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-09-15T04:56:23.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:23.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-09-15T04:56:24.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-09-15T04:56:24.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-09-15T04:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-09-15T04:56:24.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-09-15T04:56:24.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-09-15T04:56:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-09-15T04:56:24.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-09-15T04:56:24.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-09-15T04:56:24.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-09-15T04:56:24.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-09-15T04:56:24.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-09-15T04:56:24.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:24.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-09-15T04:56:25.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-09-15T04:56:25.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-09-15T04:56:25.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-09-15T04:56:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-09-15T04:56:25.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-09-15T04:56:25.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-09-15T04:56:25.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-09-15T04:56:25.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-09-15T04:56:25.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-09-15T04:56:25.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-09-15T04:56:25.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-09-15T04:56:25.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-09-15T04:56:25.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:25.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-09-15T04:56:26.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-09-15T04:56:26.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-09-15T04:56:26.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-09-15T04:56:26.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-09-15T04:56:26.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-09-15T04:56:26.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-09-15T04:56:26.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-09-15T04:56:26.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-09-15T04:56:26.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-09-15T04:56:26.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-09-15T04:56:26.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-09-15T04:56:26.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:26.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-09-15T04:56:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-09-15T04:56:27.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-09-15T04:56:27.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-09-15T04:56:27.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-09-15T04:56:27.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-09-15T04:56:27.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-09-15T04:56:27.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-09-15T04:56:27.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-09-15T04:56:27.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-09-15T04:56:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-09-15T04:56:27.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-09-15T04:56:27.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:27.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-09-15T04:56:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-09-15T04:56:28.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-09-15T04:56:28.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-09-15T04:56:28.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-09-15T04:56:28.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-09-15T04:56:28.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-09-15T04:56:28.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-09-15T04:56:28.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-09-15T04:56:28.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-09-15T04:56:28.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-09-15T04:56:28.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-09-15T04:56:28.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-09-15T04:56:28.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:28.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-09-15T04:56:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-09-15T04:56:29.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-09-15T04:56:29.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-09-15T04:56:29.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-09-15T04:56:29.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-09-15T04:56:29.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-09-15T04:56:29.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-09-15T04:56:29.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-09-15T04:56:29.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-09-15T04:56:29.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-09-15T04:56:29.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-09-15T04:56:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:29.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-09-15T04:56:30.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-09-15T04:56:30.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-09-15T04:56:30.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-09-15T04:56:30.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-09-15T04:56:30.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-09-15T04:56:30.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-09-15T04:56:30.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-09-15T04:56:30.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-09-15T04:56:30.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-09-15T04:56:30.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-09-15T04:56:30.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-09-15T04:56:30.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:30.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-09-15T04:56:31.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-09-15T04:56:31.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-09-15T04:56:31.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-09-15T04:56:31.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-09-15T04:56:31.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-09-15T04:56:31.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-09-15T04:56:31.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-09-15T04:56:31.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-09-15T04:56:31.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-09-15T04:56:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-09-15T04:56:31.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-09-15T04:56:31.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:31.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-09-15T04:56:32.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-09-15T04:56:32.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-09-15T04:56:32.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-09-15T04:56:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-09-15T04:56:32.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-09-15T04:56:32.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-09-15T04:56:32.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-09-15T04:56:32.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-09-15T04:56:32.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-09-15T04:56:32.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-09-15T04:56:32.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-09-15T04:56:32.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:32.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-09-15T04:56:33.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-09-15T04:56:33.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-09-15T04:56:33.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-09-15T04:56:33.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-09-15T04:56:33.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-09-15T04:56:33.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-09-15T04:56:33.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-09-15T04:56:33.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-09-15T04:56:33.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-09-15T04:56:33.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-09-15T04:56:33.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-09-15T04:56:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-09-15T04:56:33.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:33.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-09-15T04:56:34.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-09-15T04:56:34.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-09-15T04:56:34.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-09-15T04:56:34.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-09-15T04:56:34.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-09-15T04:56:34.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-09-15T04:56:34.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-09-15T04:56:34.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-09-15T04:56:34.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-09-15T04:56:34.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-09-15T04:56:34.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-09-15T04:56:34.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:34.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-09-15T04:56:35.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-09-15T04:56:35.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-09-15T04:56:35.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-09-15T04:56:35.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-09-15T04:56:35.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-09-15T04:56:35.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-09-15T04:56:35.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-09-15T04:56:35.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-09-15T04:56:35.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-09-15T04:56:35.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-09-15T04:56:35.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-09-15T04:56:35.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-09-15T04:56:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-09-15T04:56:36.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-09-15T04:56:36.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-09-15T04:56:36.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-09-15T04:56:36.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-09-15T04:56:36.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-09-15T04:56:36.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-09-15T04:56:36.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-09-15T04:56:36.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-09-15T04:56:36.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-09-15T04:56:36.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-09-15T04:56:36.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-09-15T04:56:36.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:36.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-09-15T04:56:37.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-09-15T04:56:37.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-09-15T04:56:37.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-09-15T04:56:37.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-09-15T04:56:37.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-09-15T04:56:37.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-09-15T04:56:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-09-15T04:56:37.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-09-15T04:56:37.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-09-15T04:56:37.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-09-15T04:56:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-09-15T04:56:37.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:37.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-09-15T04:56:38.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-09-15T04:56:38.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-09-15T04:56:38.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-09-15T04:56:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-09-15T04:56:38.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-09-15T04:56:38.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-09-15T04:56:38.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-09-15T04:56:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-09-15T04:56:38.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-09-15T04:56:38.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-09-15T04:56:38.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-09-15T04:56:38.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-09-15T04:56:38.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:38.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-09-15T04:56:39.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-09-15T04:56:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-09-15T04:56:39.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-09-15T04:56:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-09-15T04:56:39.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-09-15T04:56:39.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-09-15T04:56:39.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-09-15T04:56:39.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-09-15T04:56:39.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-09-15T04:56:39.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-09-15T04:56:39.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-09-15T04:56:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-09-15T04:56:40.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-09-15T04:56:40.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-09-15T04:56:40.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-09-15T04:56:40.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-09-15T04:56:40.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-09-15T04:56:40.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-09-15T04:56:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-09-15T04:56:40.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-09-15T04:56:40.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-09-15T04:56:40.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-09-15T04:56:40.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-09-15T04:56:40.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:40.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-09-15T04:56:40.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-09-15T04:56:41.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-09-15T04:56:41.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-09-15T04:56:41.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-09-15T04:56:41.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-09-15T04:56:41.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-09-15T04:56:41.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-09-15T04:56:41.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-09-15T04:56:41.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-09-15T04:56:41.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-09-15T04:56:41.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-09-15T04:56:41.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-09-15T04:56:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-09-15T04:56:42.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-09-15T04:56:42.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-09-15T04:56:42.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-09-15T04:56:42.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-09-15T04:56:42.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-09-15T04:56:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-09-15T04:56:42.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-09-15T04:56:42.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-09-15T04:56:42.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-09-15T04:56:42.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-09-15T04:56:42.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-09-15T04:56:42.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:42.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-09-15T04:56:43.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-09-15T04:56:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-09-15T04:56:43.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-09-15T04:56:43.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-09-15T04:56:43.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-09-15T04:56:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-09-15T04:56:43.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-09-15T04:56:43.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-09-15T04:56:43.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-09-15T04:56:43.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-09-15T04:56:43.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-09-15T04:56:43.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:43.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-09-15T04:56:44.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-09-15T04:56:44.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-09-15T04:56:44.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-09-15T04:56:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-09-15T04:56:44.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-09-15T04:56:44.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-09-15T04:56:44.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:44.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-09-15T04:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-09-15T04:56:45.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-09-15T04:56:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-09-15T04:56:45.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-09-15T04:56:45.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-09-15T04:56:45.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-09-15T04:56:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-09-15T04:56:45.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-09-15T04:56:45.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-09-15T04:56:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-09-15T04:56:45.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:45.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-09-15T04:56:46.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-09-15T04:56:46.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-09-15T04:56:46.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-09-15T04:56:46.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-09-15T04:56:46.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-09-15T04:56:46.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-09-15T04:56:46.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-09-15T04:56:46.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-09-15T04:56:46.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-09-15T04:56:46.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-09-15T04:56:46.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-09-15T04:56:46.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-09-15T04:56:46.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:46.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-09-15T04:56:47.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-09-15T04:56:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-09-15T04:56:47.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-09-15T04:56:47.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-09-15T04:56:47.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-09-15T04:56:47.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-09-15T04:56:47.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-09-15T04:56:47.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:119: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-09-15T04:56:47.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:121: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-09-15T04:56:47.767 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: expr 2 - 1 2024-09-15T04:56:47.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: seq 0 1 2024-09-15T04:56:47.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:56:47.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-09-15T04:56:47.868 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:56:47.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-09-15T04:56:47.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-09-15T04:56:47.976 INFO:tasks.workunit.client.0.smithi138.stdout:{} 2024-09-15T04:56:47.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: sleep 5 2024-09-15T04:56:52.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-15T04:56:53.325 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:56:53.325 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-09-15T04:55:52.611501+0000 23'600 23:617 [1] 1 [1] 1 0'0 2024-09-15T04:55:49.982557+0000 0'0 2024-09-15T04:55:49.982557+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:57:26.416358+0000 0 0 2024-09-15T04:56:53.325 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:56:53.325 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:56:53.325 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:56:53.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:132: TEST_recovery_test_simple: seq 1 1 2024-09-15T04:56:53.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:132: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-09-15T04:56:53.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-09-15T04:56:54.030 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T04:56:54.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:138: TEST_recovery_test_simple: wait_for_recovery_toofull 120 2024-09-15T04:56:54.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=120 2024-09-15T04:56:54.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 120 2024-09-15T04:56:54.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-09-15T04:56:54.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-09-15T04:56:54.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 120 5 2024-09-15T04:56:54.042 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T04:56:54.043 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T04:56:54.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T04:56:54.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T04:56:54.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15' '15' '15' '15' '15' '15' '15') 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T04:56:54.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T04:56:54.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T04:56:54.683 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T04:56:54.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T04:56:54.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:56:54.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T04:56:54.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836511 2024-09-15T04:56:54.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836511 2024-09-15T04:56:54.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511' 2024-09-15T04:56:54.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T04:56:54.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836511 1-42949672982' 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836511 2024-09-15T04:56:54.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:56:54.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T04:56:54.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836511 2024-09-15T04:56:54.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:56:54.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836511 2024-09-15T04:56:54.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-15T04:56:54.912 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836511 2024-09-15T04:56:54.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:56:55.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-15T04:56:55.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:56:56.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T04:56:56.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:56:56.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-15T04:56:56.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T04:56:57.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T04:56:57.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T04:56:57.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-15T04:56:57.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T04:56:57.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-15T04:56:57.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T04:56:57.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T04:56:57.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-15T04:56:57.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T04:56:57.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-15T04:56:57.975 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672982 2024-09-15T04:56:57.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-15T04:56:57.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T04:56:58.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672983 -lt 42949672982 2024-09-15T04:56:58.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-09-15T04:56:58.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T04:56:58.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T04:56:58.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-09-15T04:56:58.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-09-15T04:56:58.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-09-15T04:56:59.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-09-15T04:56:59.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:79: wait_for_recovery_toofull: '[' -ne 0 ']' 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh: line 79: [: -ne: unary operator expected 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:139: TEST_recovery_test_simple: '[' 0 -ne 0 ']' 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ERRORS=0 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: grep +recovery_toofull 2024-09-15T04:56:59.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: wc -l 2024-09-15T04:56:59.496 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:56:59.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-09-15T04:56:59.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:151: TEST_recovery_test_simple: ceph pg dump pgs 2024-09-15T04:56:59.838 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T04:56:59.838 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-09-15T04:56:54.642344+0000 23'600 27:631 [1,0] 1 [1] 1 0'0 2024-09-15T04:55:49.982557+0000 0'0 2024-09-15T04:55:49.982557+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:03:31.611961+0000 0 0 2024-09-15T04:56:59.838 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:56:59.838 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T04:56:59.839 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T04:56:59.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: ceph status 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: cluster: 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: id: c2a70de4-d034-48c4-a126-d95d1494073f 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: health: HEALTH_ERR 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: 2 full osd(s) 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: 1 pool(s) do not have an application enabled 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: 1 pool(s) full 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: OSD count 2 < osd_pool_default_size 3 2024-09-15T04:57:00.316 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: services: 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: mon: 1 daemons, quorum a (age 2m) 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: mgr: x(active, since 2m) 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: osd: 2 osds: 2 up (since 74s), 2 in (since 107s); 1 remapped pgs 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: data: 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: pools: 1 pools, 1 pgs 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: objects: 600 objects, 2.9 MiB 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: usage: 5.9 MiB used, 1.2 MiB / 7.0 MiB avail 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-09-15T04:57:00.317 INFO:tasks.workunit.client.0.smithi138.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-09-15T04:57:00.318 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T04:57:00.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-09-15T04:57:00.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-09-15T04:57:00.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-09-15T04:57:00.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:155: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-09-15T04:57:00.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:156: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-09-15T04:57:00.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-09-15T04:57:00.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-09-15T04:57:00.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:160: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-09-15T04:57:00.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:161: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-09-15T04:57:00.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:165: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-09-15T04:57:00.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:167: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-09-15T04:57:00.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:172: TEST_recovery_test_simple: seq 1 1 2024-09-15T04:57:00.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:172: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-09-15T04:57:00.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:174: TEST_recovery_test_simple: delete_pool test1 2024-09-15T04:57:00.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-09-15T04:57:00.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-09-15T04:57:01.287 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test1' does not exist 2024-09-15T04:57:01.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:176: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-09-15T04:57:01.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:57:01.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:57:01.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:57:01.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:57:01.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:57:06.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:57:06.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:57:06.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:57:06.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:57:06.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:57:06.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:57:06.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:57:06.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:57:06.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:57:06.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:57:06.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:57:06.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:57:06.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:57:06.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:57:06.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-15T04:57:06.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:57:06.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:06.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:57:06.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.273887 2024-09-15T04:57:06.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:57:06.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:57:06.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T04:57:06.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-space 0 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:57:06.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:57:06.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:57:06.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:57:06.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:57:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:57:06.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:57:06.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:57:06.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:57:06.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:57:06.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:57:06.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T04:57:06.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-09-15T04:57:06.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:57:06.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:06.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.273887 2024-09-15T04:57:06.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.273887 2024-09-15T04:57:06.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:57:06.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:57:06.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T04:57:06.669 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T04:57:06.669 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T04:57:06.695 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-09-15T04:57:06.695 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-09-15T04:57:06.759 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T04:57:06.765 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-recovery-stats 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-09-15T04:57:06.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none ' 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-09-15T04:57:06.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-09-15T04:57:06.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-09-15T04:57:06.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-09-15T04:57:06.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-09-15T04:57:06.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-09-15T04:57:06.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_erasure_remapped 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_multi 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_replicated_out1 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_sizedown 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_sizeup 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_recovery_undersized' 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T04:57:06.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T04:57:06.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T04:57:06.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T04:57:06.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T04:57:06.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T04:57:06.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T04:57:06.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T04:57:06.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T04:57:06.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T04:57:06.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T04:57:06.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T04:57:06.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T04:57:06.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T04:57:06.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T04:57:06.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:06.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:06.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T04:57:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T04:57:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T04:57:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T04:57:06.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T04:57:06.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:06.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-09-15T04:57:06.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T04:57:06.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T04:57:07.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T04:57:07.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:07.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T04:57:07.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:07.406 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T04:57:07.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:57:07.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T04:57:07.479 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T04:57:07.480 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:07.480 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:07.480 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T04:57:07.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T04:57:07.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T04:57:07.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T04:57:07.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T04:57:08.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:57:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T04:57:08.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-15T04:57:08.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T04:57:08.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:57:08.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T04:57:08.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:57:08.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T04:57:08.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:57:08.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae 2024-09-15T04:57:08.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae' 2024-09-15T04:57:08.412 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae 2024-09-15T04:57:08.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:57:08.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkaeZmEQUuGRAARzCtAnOubOYFN97Mc6EK5g== 2024-09-15T04:57:08.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkaeZmEQUuGRAARzCtAnOubOYFN97Mc6EK5g=="}' 2024-09-15T04:57:08.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae -i td/osd-recovery-stats/0/new.json 2024-09-15T04:57:08.742 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:57:08.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T04:57:08.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkaeZmEQUuGRAARzCtAnOubOYFN97Mc6EK5g== --osd-uuid 4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae 2024-09-15T04:57:08.777 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.776+0000 7fb890321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:08.788 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.787+0000 7fb890321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:08.789 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.788+0000 7fb890321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:08.789 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.788+0000 7fb890321440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T04:57:08.789 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.788+0000 7fb890321440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T04:57:08.789 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.788+0000 7fb890321440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T04:57:08.828 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:08.827+0000 7fb890321440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T04:57:14.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T04:57:14.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:57:14.208 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T04:57:14.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T04:57:14.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:57:14.696 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T04:57:14.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T04:57:14.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:57:14.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:57:14.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:57:14.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:57:14.719 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:14.718+0000 7f8f10609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:14.720+0000 7f8f10609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:14.722 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:14.722+0000 7f8f10609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:57:15.047 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:57:15.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:15.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:15.245 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:15.244+0000 7f8f10609440 -1 Falling back to public interface 2024-09-15T04:57:15.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:16.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:16.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:16.407 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:57:16.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:57:16.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:16.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:16.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:17.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:17.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:17.757 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:57:17.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:57:17.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:17.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:17.763 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:17.762+0000 7f8f10609440 -1 osd.0 0 log_to_monitors true 2024-09-15T04:57:18.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:19.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:19.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:19.137 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:57:19.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:57:19.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:19.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:19.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:20.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:20.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:20.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:57:20.645 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:57:20.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:20.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:21.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:22.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:22.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:22.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:57:22.034 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:57:22.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:22.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:22.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:23.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:23.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:23.441 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:57:23.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:57:23.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:23.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:23.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:24.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:24.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:24.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:57:24.801 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:57:24.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:24.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:25.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:26.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:26.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:26.163 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:57:26.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:57:26.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:26.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:26.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:27.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:27.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:27.548 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:57:27.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:57:27.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:27.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:27.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:28.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:28.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:28.947 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:57:28.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:57:28.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:28.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:29.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:30.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:30.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:30.506 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:57:30.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:57:30.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:30.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:31.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:32.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:32.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:32.022 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:57:32.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:57:32.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:32.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:32.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:57:33.756 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:57:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:34.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:35.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:35.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:35.297 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:57:35.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:57:35.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:35.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:35.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:36.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:36.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:36.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:57:36.999 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:57:36.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:36.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:37.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:38.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:38.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:38.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:57:38.632 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:57:38.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:38.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:39.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:40.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:40.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:40.217 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:57:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:57:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:40.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T04:57:40.638 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/291753,v1:127.0.0.1:6803/291753] [v2:127.0.0.1:6804/291753,v1:127.0.0.1:6805/291753] exists,up 4e08fa2a-1f51-4261-8e9e-bfa7c4ad34ae 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:57:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:57:40.643 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:57:40.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:57:40.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:57:40.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T04:57:40.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:57:40.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7f709b62-61fb-4fb7-8baf-e25dcd33238e 2024-09-15T04:57:40.646 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 7f709b62-61fb-4fb7-8baf-e25dcd33238e 2024-09-15T04:57:40.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7f709b62-61fb-4fb7-8baf-e25dcd33238e' 2024-09-15T04:57:40.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:57:40.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBEaeZmGBQOJxAA6JBRrtfS3P3HQbE+cHGJfw== 2024-09-15T04:57:40.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBEaeZmGBQOJxAA6JBRrtfS3P3HQbE+cHGJfw=="}' 2024-09-15T04:57:40.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7f709b62-61fb-4fb7-8baf-e25dcd33238e -i td/osd-recovery-stats/1/new.json 2024-09-15T04:57:41.161 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:57:41.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T04:57:41.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEaeZmGBQOJxAA6JBRrtfS3P3HQbE+cHGJfw== --osd-uuid 7f709b62-61fb-4fb7-8baf-e25dcd33238e 2024-09-15T04:57:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.195+0000 7f21c5723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:41.199 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.197+0000 7f21c5723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.198+0000 7f21c5723440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.198+0000 7f21c5723440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T04:57:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.198+0000 7f21c5723440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T04:57:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.198+0000 7f21c5723440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T04:57:41.252 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:41.250+0000 7f21c5723440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T04:57:47.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T04:57:47.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:57:47.669 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T04:57:47.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T04:57:47.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:57:48.155 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T04:57:48.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T04:57:48.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:57:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:57:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:57:48.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:57:48.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:48.176+0000 7f8026d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:48.179 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:48.179+0000 7f8026d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:48.180 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:48.180+0000 7f8026d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:57:48.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:57:48.505 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:57:48.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:48.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:48.704 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:48.703+0000 7f8026d1f440 -1 Falling back to public interface 2024-09-15T04:57:48.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:49.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:49.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:49.857 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:57:49.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:57:49.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:49.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:50.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:51.183 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:57:51.182+0000 7f8026d1f440 -1 osd.1 0 log_to_monitors true 2024-09-15T04:57:51.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:51.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:51.236 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:57:51.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:57:51.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:51.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:51.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:52.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:52.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:52.945 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:57:52.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:57:52.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:52.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:53.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:54.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:54.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:54.391 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:57:54.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:57:54.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:54.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:54.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:55.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:55.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:55.889 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:57:55.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:57:55.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:55.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:56.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:57.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:57.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:57.333 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:57:57.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:57:57.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:57.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:57.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:57:58.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:57:58.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:57:58.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:57:58.728 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:57:58.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:57:58.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:57:59.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:00.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:00.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:00.257 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:58:00.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:58:00.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:00.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:00.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:01.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:01.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:01.746 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:58:01.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:58:01.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:01.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:02.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:03.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:03.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:03.183 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:58:03.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:58:03.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:03.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:03.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:04.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:04.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:04.571 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:58:04.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:58:04.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:04.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:05.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:06.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:06.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:06.183 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:58:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:58:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:06.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:07.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:07.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:07.924 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:58:07.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:58:07.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:07.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:08.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:09.693 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:58:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:58:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:10.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:11.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:11.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:11.436 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:58:11.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:58:11.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:11.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:12.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:13.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:13.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:13.253 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:58:13.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:58:13.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:13.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:14.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:14.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:14.992 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:58:14.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:58:14.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:14.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T04:58:15.413 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/293507,v1:127.0.0.1:6811/293507] [v2:127.0.0.1:6812/293507,v1:127.0.0.1:6813/293507] exists,up 7f709b62-61fb-4fb7-8baf-e25dcd33238e 2024-09-15T04:58:15.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:58:15.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T04:58:15.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:58:15.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0 2024-09-15T04:58:15.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0' 2024-09-15T04:58:15.421 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0 2024-09-15T04:58:15.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:58:15.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBnaeZmUgupGRAAd4bllNTNA1dYVabi+xA4wA== 2024-09-15T04:58:15.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBnaeZmUgupGRAAd4bllNTNA1dYVabi+xA4wA=="}' 2024-09-15T04:58:15.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0 -i td/osd-recovery-stats/2/new.json 2024-09-15T04:58:15.926 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:58:15.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T04:58:15.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBnaeZmUgupGRAAd4bllNTNA1dYVabi+xA4wA== --osd-uuid 9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0 2024-09-15T04:58:15.962 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.961+0000 7f1695832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:15.963 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.963+0000 7f1695832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:15.964 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.964+0000 7f1695832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:15.965 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.964+0000 7f1695832440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T04:58:15.965 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.964+0000 7f1695832440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T04:58:15.965 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:15.964+0000 7f1695832440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T04:58:16.033 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:16.032+0000 7f1695832440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T04:58:22.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T04:58:22.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:58:22.366 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T04:58:22.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T04:58:22.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:58:22.855 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T04:58:22.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T04:58:22.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:58:22.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:58:22.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:58:22.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:58:22.880 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:22.877+0000 7f72eb91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:22.880 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:22.879+0000 7f72eb91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:22.881 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:22.880+0000 7f72eb91f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:23.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T04:58:23.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:58:23.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:23.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:23.398 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:23.397+0000 7f72eb91f440 -1 Falling back to public interface 2024-09-15T04:58:23.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:24.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:24.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:24.553 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:58:24.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:58:24.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:24.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:24.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:25.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:25.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:25.900 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:58:25.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:58:25.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:25.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:25.905 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:25.903+0000 7f72eb91f440 -1 osd.2 0 log_to_monitors true 2024-09-15T04:58:26.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:27.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:27.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:27.288 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:58:27.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:58:27.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:27.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:27.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:28.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:28.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:28.672 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:58:28.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:58:28.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:28.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:29.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:30.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:58:30.079 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:58:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:30.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:30.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:31.439 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:58:31.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:58:31.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:31.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:31.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:32.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:32.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:32.924 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:58:32.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:58:32.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:32.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:33.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:34.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:34.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:34.453 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:58:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:58:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:34.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:35.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:35.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:35.937 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:58:35.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:58:35.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:35.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:36.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:37.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:37.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:37.333 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:58:37.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:58:37.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:37.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:37.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:38.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:38.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:38.850 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:58:38.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:58:38.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:38.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:39.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:40.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:40.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:40.550 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:58:40.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:58:40.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:40.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:41.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:42.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:42.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:42.148 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:58:42.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:58:42.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:42.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:42.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:43.733 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:58:43.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:58:43.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:43.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:44.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:45.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:45.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:45.442 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:58:45.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:58:45.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:45.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:46.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:47.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:47.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:47.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:58:47.196 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:58:47.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:47.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:47.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:48.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:48.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:48.983 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:58:48.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:58:48.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:48.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:49.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:50.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:50.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:50.453 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:58:50.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:58:50.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:50.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T04:58:50.823 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/295261,v1:127.0.0.1:6819/295261] [v2:127.0.0.1:6820/295261,v1:127.0.0.1:6821/295261] exists,up 9e6e773d-c8ef-4cc3-bbf9-e4b03b2dc7d0 2024-09-15T04:58:50.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:58:50.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:58:50.829 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:58:50.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f540beb6-1aa9-49ce-aaed-6ea08a83d6e1 2024-09-15T04:58:50.831 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 f540beb6-1aa9-49ce-aaed-6ea08a83d6e1 2024-09-15T04:58:50.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f540beb6-1aa9-49ce-aaed-6ea08a83d6e1' 2024-09-15T04:58:50.832 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:58:50.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCKaeZmN08hMhAAgQ+dSuFg27pEp0KLq558eQ== 2024-09-15T04:58:50.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCKaeZmN08hMhAAgQ+dSuFg27pEp0KLq558eQ=="}' 2024-09-15T04:58:50.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f540beb6-1aa9-49ce-aaed-6ea08a83d6e1 -i td/osd-recovery-stats/3/new.json 2024-09-15T04:58:51.289 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:58:51.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T04:58:51.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCKaeZmN08hMhAAgQ+dSuFg27pEp0KLq558eQ== --osd-uuid f540beb6-1aa9-49ce-aaed-6ea08a83d6e1 2024-09-15T04:58:51.325 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.324+0000 7fb94c487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:51.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.359+0000 7fb94c487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:51.361 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.360+0000 7fb94c487440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.361+0000 7fb94c487440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T04:58:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.361+0000 7fb94c487440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T04:58:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.361+0000 7fb94c487440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T04:58:51.433 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:51.432+0000 7fb94c487440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T04:58:57.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T04:58:57.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:58:57.608 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T04:58:57.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T04:58:57.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:58:58.112 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T04:58:58.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T04:58:58.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:58:58.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:58:58.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:58:58.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:58:58.134 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:58.133+0000 7fc1cfa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:58.136 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:58.135+0000 7fc1cfa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:58.137 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:58.136+0000 7fc1cfa08440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:58.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:58:58.463 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:58:58.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:58.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:58:58.652 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:58:58.651+0000 7fc1cfa08440 -1 Falling back to public interface 2024-09-15T04:58:58.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:58:59.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:58:59.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:58:59.816 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:58:59.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:58:59.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:58:59.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:00.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:01.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:01.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:01.166 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:59:01.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:59:01.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:01.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:01.188 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:01.187+0000 7fc1cfa08440 -1 osd.3 0 log_to_monitors true 2024-09-15T04:59:01.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:02.666 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:59:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:59:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:02.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:03.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:04.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:04.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:04.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:59:04.181 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:59:04.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:04.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:04.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:05.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:05.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:05.540 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:59:05.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:59:05.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:05.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:06.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:07.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:07.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:07.002 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:59:07.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:59:07.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:07.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:07.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:08.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:08.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:08.432 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:59:08.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:59:08.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:08.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:08.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:09.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:09.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:09.905 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:59:09.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:59:09.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:09.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:10.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:11.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:11.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:11.294 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:59:11.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:59:11.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:11.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:11.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:12.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:12.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:12.703 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:59:12.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:59:12.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:12.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:13.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:14.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:14.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:14.207 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:59:14.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:59:14.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:14.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:14.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:15.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:15.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:15.689 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:59:15.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:59:15.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:15.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:16.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:17.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:17.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:17.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:59:17.881 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:59:17.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:17.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:18.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:19.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:19.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:19.669 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:59:19.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:59:19.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:19.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:20.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:21.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:21.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:21.348 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:59:21.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:59:21.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:21.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:22.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:23.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:23.182 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:59:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:59:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:23.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:24.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:25.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:25.002 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T04:59:25.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T04:59:25.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:25.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:25.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:26.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:26.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:26.602 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T04:59:26.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T04:59:26.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:26.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T04:59:26.961 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/297040,v1:127.0.0.1:6827/297040] [v2:127.0.0.1:6828/297040,v1:127.0.0.1:6829/297040] exists,up f540beb6-1aa9-49ce-aaed-6ea08a83d6e1 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T04:59:26.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T04:59:26.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T04:59:26.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-15T04:59:26.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T04:59:26.970 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 906a54e5-70c6-4f3e-9ad5-fd6c5d59c802 2024-09-15T04:59:26.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=906a54e5-70c6-4f3e-9ad5-fd6c5d59c802 2024-09-15T04:59:26.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 906a54e5-70c6-4f3e-9ad5-fd6c5d59c802' 2024-09-15T04:59:26.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T04:59:26.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCuaeZmzQ1XOhAAkEDBIennY47KGG3es74jjg== 2024-09-15T04:59:26.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCuaeZmzQ1XOhAAkEDBIennY47KGG3es74jjg=="}' 2024-09-15T04:59:26.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 906a54e5-70c6-4f3e-9ad5-fd6c5d59c802 -i td/osd-recovery-stats/4/new.json 2024-09-15T04:59:27.429 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:59:27.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-15T04:59:27.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCuaeZmzQ1XOhAAkEDBIennY47KGG3es74jjg== --osd-uuid 906a54e5-70c6-4f3e-9ad5-fd6c5d59c802 2024-09-15T04:59:27.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.463+0000 7f4167289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:27.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.480+0000 7f4167289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.481+0000 7f4167289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:27.482 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.482+0000 7f4167289440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T04:59:27.483 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.482+0000 7f4167289440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T04:59:27.483 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.482+0000 7f4167289440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T04:59:27.521 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:27.520+0000 7f4167289440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-15T04:59:33.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-15T04:59:33.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T04:59:33.731 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T04:59:33.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T04:59:33.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T04:59:34.214 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T04:59:34.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T04:59:34.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T04:59:34.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T04:59:34.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T04:59:34.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T04:59:34.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:34.236+0000 7f0b3562f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:34.240 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:34.238+0000 7f0b3562f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:34.240 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:34.239+0000 7f0b3562f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:34.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T04:59:34.579 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T04:59:34.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:34.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:34.759 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:34.758+0000 7f0b3562f440 -1 Falling back to public interface 2024-09-15T04:59:34.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:35.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:35.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:35.932 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T04:59:35.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T04:59:35.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:35.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:36.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:37.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:37.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:37.290 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T04:59:37.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T04:59:37.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:37.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:37.295 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T04:59:37.294+0000 7f0b3562f440 -1 osd.4 0 log_to_monitors true 2024-09-15T04:59:37.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:38.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:38.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:38.773 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T04:59:38.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T04:59:38.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:38.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:39.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:40.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:40.232 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T04:59:40.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T04:59:40.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:40.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:40.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:41.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:41.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:41.830 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T04:59:41.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T04:59:41.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:41.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:42.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:43.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:43.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:43.302 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T04:59:43.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T04:59:43.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:43.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:43.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:44.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:44.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:44.700 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T04:59:44.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T04:59:44.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:44.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:45.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:46.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:46.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T04:59:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T04:59:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:46.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:47.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:47.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:47.521 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T04:59:47.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T04:59:47.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:47.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:47.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:48.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:48.895 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T04:59:48.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:48.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T04:59:48.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:48.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:49.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:50.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:50.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:50.418 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T04:59:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T04:59:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:51.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:52.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:52.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:52.081 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T04:59:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T04:59:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:52.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:53.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:53.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:53.548 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T04:59:53.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T04:59:53.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:53.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:54.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:55.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:55.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:55.328 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T04:59:55.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T04:59:55.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:55.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:56.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:57.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:57.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:57.157 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T04:59:57.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T04:59:57.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:57.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:58.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T04:59:59.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T04:59:59.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T04:59:59.212 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T04:59:59.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T04:59:59.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T04:59:59.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T04:59:59.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:00.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:00.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:00.943 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:00:00.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:00:00.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:00.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:00:01.402 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/298820,v1:127.0.0.1:6835/298820] [v2:127.0.0.1:6836/298820,v1:127.0.0.1:6837/298820] exists,up 906a54e5-70c6-4f3e-9ad5-fd6c5d59c802 2024-09-15T05:00:01.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:00:01.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:00:01.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3b9d0ce4-26b8-4594-987b-4bfa41e2fc42 2024-09-15T05:00:01.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 3b9d0ce4-26b8-4594-987b-4bfa41e2fc42' 2024-09-15T05:00:01.410 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 3b9d0ce4-26b8-4594-987b-4bfa41e2fc42 2024-09-15T05:00:01.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:00:01.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDRaeZmn1QOGRAA9E/dqLmJajbFsjVrIWqFCw== 2024-09-15T05:00:01.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDRaeZmn1QOGRAA9E/dqLmJajbFsjVrIWqFCw=="}' 2024-09-15T05:00:01.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3b9d0ce4-26b8-4594-987b-4bfa41e2fc42 -i td/osd-recovery-stats/5/new.json 2024-09-15T05:00:01.854 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:00:01.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-15T05:00:01.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDRaeZmn1QOGRAA9E/dqLmJajbFsjVrIWqFCw== --osd-uuid 3b9d0ce4-26b8-4594-987b-4bfa41e2fc42 2024-09-15T05:00:01.889 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.887+0000 7f0228a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:01.939 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.937+0000 7f0228a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:01.940 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.938+0000 7f0228a67440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:01.940 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.938+0000 7f0228a67440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:00:01.940 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.938+0000 7f0228a67440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:00:01.940 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.938+0000 7f0228a67440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:00:01.984 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:01.983+0000 7f0228a67440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-15T05:00:08.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-15T05:00:08.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:00:08.051 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:00:08.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:00:08.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:00:08.542 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:00:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:00:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:00:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:00:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:00:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:00:08.565 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:08.564+0000 7f47dda2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:08.567 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:08.566+0000 7f47dda2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:08.568 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:08.567+0000 7f47dda2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:00:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:08.900 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:00:08.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:00:08.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:08.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:09.085 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:09.084+0000 7f47dda2d440 -1 Falling back to public interface 2024-09-15T05:00:09.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:10.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:10.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:10.288 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:00:10.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:00:10.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:10.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:10.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:11.569 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:00:11.568+0000 7f47dda2d440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:00:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:11.685 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:00:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:00:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:12.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:13.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:13.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:13.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:00:13.280 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:00:13.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:13.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:13.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:14.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:14.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:14.641 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:00:14.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:00:14.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:14.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:15.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:16.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:16.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:16.044 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:00:16.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:00:16.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:16.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:16.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:17.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:17.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:17.474 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:00:17.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:00:17.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:17.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:17.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:18.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:18.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:18.912 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:00:18.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:00:18.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:18.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:19.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:20.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:20.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:20.355 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:00:20.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:00:20.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:20.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:20.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:21.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:21.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:21.715 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:00:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:00:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:21.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:22.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:23.078 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:00:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:00:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:23.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:24.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:24.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:24.496 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:00:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:00:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:24.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:25.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:26.192 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:00:26.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:00:26.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:26.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:27.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:28.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:28.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:28.006 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:00:28.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:00:28.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:28.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:28.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:29.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:29.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:29.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:00:29.731 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:00:29.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:29.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:30.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:31.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:31.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:31.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:00:31.150 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:00:31.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:31.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:31.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:32.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:32.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:32.879 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:00:32.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:00:32.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:32.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:33.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:34.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:34.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:34.735 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:00:34.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:00:34.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:34.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:35.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:00:36.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:00:36.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:00:36.363 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:00:36.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:00:36.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:00:36.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:00:36.790 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/300575,v1:127.0.0.1:6843/300575] [v2:127.0.0.1:6844/300575,v1:127.0.0.1:6845/300575] exists,up 3b9d0ce4-26b8-4594-987b-4bfa41e2fc42 2024-09-15T05:00:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:00:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:00:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:00:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-09-15T05:00:36.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T05:00:37.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-09-15T05:00:37.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-15T05:00:38.498 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:00:38.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:00:39.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-15T05:00:39.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:00:39.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:00:39.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:00:39.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:00:39.512 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:00:39.512 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:00:39.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:00:39.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:00:39.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:00:39.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:00:39.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:00:39.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:00:39.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:00:39.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:00:39.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:00:39.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:00:39.959 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:00:39.959 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:00:39.959 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:00:39.959 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:00:39.960 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:00:39.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:00:39.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:39.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:00:40.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-15T05:00:40.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-15T05:00:40.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-15T05:00:40.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:40.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:00:40.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672992 2024-09-15T05:00:40.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672992 2024-09-15T05:00:40.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672992' 2024-09-15T05:00:40.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:40.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:00:40.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509468 2024-09-15T05:00:40.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509468 2024-09-15T05:00:40.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672992 2-64424509468' 2024-09-15T05:00:40.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:40.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:00:40.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-15T05:00:40.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-15T05:00:40.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672992 2-64424509468 3-85899345939' 2024-09-15T05:00:40.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:40.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:00:40.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182414 2024-09-15T05:00:40.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182414 2024-09-15T05:00:40.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672992 2-64424509468 3-85899345939 4-107374182414' 2024-09-15T05:00:40.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:00:40.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:00:40.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018889 2024-09-15T05:00:40.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018889 2024-09-15T05:00:40.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672992 2-64424509468 3-85899345939 4-107374182414 5-128849018889' 2024-09-15T05:00:40.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:40.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-15T05:00:40.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:40.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:00:40.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-15T05:00:40.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:40.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-15T05:00:40.763 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836521 2024-09-15T05:00:40.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-15T05:00:40.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:00:41.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-15T05:00:41.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:00:42.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:00:42.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:00:42.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836521 2024-09-15T05:00:42.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:42.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672992 2024-09-15T05:00:42.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:42.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:00:42.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672992 2024-09-15T05:00:42.508 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:42.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672992 2024-09-15T05:00:42.509 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672992 2024-09-15T05:00:42.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-15T05:00:42.509 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:00:42.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672993 -lt 42949672992 2024-09-15T05:00:42.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:42.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509468 2024-09-15T05:00:42.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:42.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:00:42.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509468 2024-09-15T05:00:42.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509468 2024-09-15T05:00:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509468' 2024-09-15T05:00:42.884 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509468 2024-09-15T05:00:42.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:00:43.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509469 -lt 64424509468 2024-09-15T05:00:43.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:43.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345939 2024-09-15T05:00:43.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:43.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:00:43.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345939 2024-09-15T05:00:43.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:43.237 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345939 2024-09-15T05:00:43.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-15T05:00:43.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345939' 2024-09-15T05:00:43.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:00:43.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345940 -lt 85899345939 2024-09-15T05:00:43.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:43.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182414 2024-09-15T05:00:43.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:43.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:00:43.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182414 2024-09-15T05:00:43.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:43.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182414 2024-09-15T05:00:43.587 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182414 2024-09-15T05:00:43.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-09-15T05:00:43.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:00:43.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182414 2024-09-15T05:00:43.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:00:43.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018889 2024-09-15T05:00:43.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:00:43.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:00:43.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018889 2024-09-15T05:00:43.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:00:43.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018889 2024-09-15T05:00:43.972 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018889 2024-09-15T05:00:43.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2024-09-15T05:00:43.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:00:44.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018890 -lt 128849018889 2024-09-15T05:00:44.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:00:44.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:00:44.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:00:44.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:00:44.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:00:44.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:00:44.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:00:44.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:00:44.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:00:44.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:00:44.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:00:45.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:00:45.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:00:45.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:00:45.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:00:45.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:00:45.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:00:45.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:00:45.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-15T05:00:45.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:45.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-15T05:00:45.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:45.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-15T05:00:45.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:45.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-15T05:00:46.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:46.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-15T05:00:46.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:46.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-15T05:00:46.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:46.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-15T05:00:46.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:46.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-15T05:00:47.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-15T05:00:47.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-15T05:00:47.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-15T05:00:47.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-15T05:00:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-15T05:00:47.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:47.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-15T05:00:48.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:48.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-15T05:00:48.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:48.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-15T05:00:48.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:48.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-15T05:00:48.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:48.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-15T05:00:48.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:48.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-15T05:00:49.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-15T05:00:49.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-15T05:00:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-15T05:00:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-15T05:00:49.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-15T05:00:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-15T05:00:50.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:50.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-15T05:00:50.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:50.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-15T05:00:50.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:50.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-15T05:00:50.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:50.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-15T05:00:50.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:50.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-15T05:00:51.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:51.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-15T05:00:51.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:51.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-15T05:00:51.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:51.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-15T05:00:51.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:51.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-15T05:00:51.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:51.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-15T05:00:52.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-15T05:00:52.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-15T05:00:52.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-15T05:00:52.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-15T05:00:52.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-15T05:00:52.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:52.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-15T05:00:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-15T05:00:53.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:53.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-15T05:00:53.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:53.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-15T05:00:53.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:53.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-15T05:00:53.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:53.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-15T05:00:54.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-15T05:00:54.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-15T05:00:54.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-15T05:00:54.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-15T05:00:54.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-15T05:00:54.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:54.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-15T05:00:55.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:55.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-15T05:00:55.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:55.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-15T05:00:55.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:55.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-15T05:00:55.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:55.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-15T05:00:55.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:55.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-15T05:00:56.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:56.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-15T05:00:56.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:56.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-15T05:00:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-15T05:00:56.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:56.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-15T05:00:56.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:56.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-15T05:00:57.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:57.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-15T05:00:57.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:57.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-15T05:00:57.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:57.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-15T05:00:57.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:57.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-15T05:00:57.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:57.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-15T05:00:58.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:58.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-15T05:00:58.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:58.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-15T05:00:58.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:58.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-15T05:00:58.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:58.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-15T05:00:58.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:58.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-15T05:00:59.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-15T05:00:59.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-15T05:00:59.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-15T05:00:59.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-15T05:00:59.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-15T05:00:59.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:00:59.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-15T05:01:00.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:00.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-15T05:01:00.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:00.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-15T05:01:00.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:00.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-15T05:01:00.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:00.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-15T05:01:00.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:00.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-15T05:01:01.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-15T05:01:01.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-15T05:01:01.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-15T05:01:01.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-15T05:01:01.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-15T05:01:01.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:01.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-15T05:01:02.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-15T05:01:02.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-15T05:01:02.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-15T05:01:02.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-15T05:01:02.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-15T05:01:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:02.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-15T05:01:03.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:03.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-15T05:01:03.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:03.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-15T05:01:03.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:03.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-15T05:01:03.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:03.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-15T05:01:03.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:03.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-15T05:01:04.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:04.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-15T05:01:04.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:04.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-15T05:01:04.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:04.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-15T05:01:04.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:04.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-15T05:01:04.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:04.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-15T05:01:05.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:05.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-15T05:01:05.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:05.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-15T05:01:05.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:05.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-15T05:01:05.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:05.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-15T05:01:05.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:05.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-15T05:01:06.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-15T05:01:06.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-15T05:01:06.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-15T05:01:06.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-15T05:01:06.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-15T05:01:06.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:06.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-15T05:01:07.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:07.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-15T05:01:07.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:07.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-15T05:01:07.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:07.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-15T05:01:07.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:07.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-15T05:01:07.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:07.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-15T05:01:08.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:08.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-15T05:01:08.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:08.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-15T05:01:08.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:08.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-15T05:01:08.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:08.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-15T05:01:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:08.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-15T05:01:09.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-15T05:01:09.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-15T05:01:09.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-15T05:01:09.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-15T05:01:09.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-15T05:01:09.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:09.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-15T05:01:10.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:10.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-15T05:01:10.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:10.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-15T05:01:10.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:10.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-15T05:01:10.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:10.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-15T05:01:10.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:10.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-15T05:01:11.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:11.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-15T05:01:11.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:11.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-15T05:01:11.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:11.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-15T05:01:11.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:11.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-15T05:01:11.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:11.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-15T05:01:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-15T05:01:12.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:12.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-15T05:01:12.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:12.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-15T05:01:12.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:12.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-15T05:01:12.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:12.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-15T05:01:13.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:13.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-15T05:01:13.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:13.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-15T05:01:13.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:13.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-15T05:01:13.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:13.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-15T05:01:13.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:13.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-15T05:01:14.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-15T05:01:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-15T05:01:14.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-15T05:01:14.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-15T05:01:14.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-15T05:01:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-15T05:01:15.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:15.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-15T05:01:15.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:15.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-15T05:01:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-15T05:01:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-15T05:01:16.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:16.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-15T05:01:16.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:16.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-15T05:01:16.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:16.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-15T05:01:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-15T05:01:16.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:16.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-15T05:01:17.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:17.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-15T05:01:17.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:17.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-15T05:01:17.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:17.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-15T05:01:17.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:17.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-15T05:01:18.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:18.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-15T05:01:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-15T05:01:18.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:18.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-15T05:01:18.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:18.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-15T05:01:18.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:18.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-15T05:01:19.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-15T05:01:19.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-15T05:01:19.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-15T05:01:19.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-15T05:01:19.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-15T05:01:19.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:19.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-15T05:01:20.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:20.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-15T05:01:20.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:20.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-15T05:01:20.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:20.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-15T05:01:20.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:20.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-15T05:01:20.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:20.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-15T05:01:21.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-15T05:01:21.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-15T05:01:21.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-15T05:01:21.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-15T05:01:21.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-15T05:01:21.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:21.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-15T05:01:22.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:22.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-15T05:01:22.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:22.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-15T05:01:22.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:22.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-15T05:01:22.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:22.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-15T05:01:22.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:22.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-15T05:01:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:23.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-15T05:01:23.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:23.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-15T05:01:23.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:01:23.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-15T05:01:23.593 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-15T05:01:23.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:01:23.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:01:23.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:01:23.594 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:01:23.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-15T05:01:23.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-15T05:01:23.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:01:23.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:01:23.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:01:23.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:01:24.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:01:24.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:01:24.315 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:01:24.315 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:01:24.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T05:01:24.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-15T05:01:24.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T05:01:25.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-15T05:01:25.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-15T05:01:25.733 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:01:25.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-15T05:01:25.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 291753 2024-09-15T05:01:25.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-15T05:01:26.247 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:01:26.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-15T05:01:26.975 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:01:26.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-15T05:01:28.487 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:01:28.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-15T05:01:28.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:01:28.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:01:28.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:01:28.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:01:28.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T05:01:29.073 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:01:29.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-15T05:01:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-15T05:01:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:01:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:01:31.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:01:31.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:01:31.084 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:01:31.084 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:01:31.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:01:31.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:01:31.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:01:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:01:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:01:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:01:31.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:01:31.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:01:31.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:01:31.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:01:31.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:31.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:01:31.670 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-15T05:01:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T05:01:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T05:01:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T05:01:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:31.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:01:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673005 2024-09-15T05:01:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673005 2024-09-15T05:01:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005' 2024-09-15T05:01:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:31.779 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:01:31.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509481 2024-09-15T05:01:31.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509481 2024-09-15T05:01:31.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509481' 2024-09-15T05:01:31.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:31.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:01:32.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345952 2024-09-15T05:01:32.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345952 2024-09-15T05:01:32.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509481 3-85899345952' 2024-09-15T05:01:32.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:32.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:01:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182427 2024-09-15T05:01:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182427 2024-09-15T05:01:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509481 3-85899345952 4-107374182427' 2024-09-15T05:01:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:01:32.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:01:32.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018902 2024-09-15T05:01:32.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018902 2024-09-15T05:01:32.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509481 3-85899345952 4-107374182427 5-128849018902' 2024-09-15T05:01:32.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:01:32.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673005 2024-09-15T05:01:32.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:01:32.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:01:32.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673005 2024-09-15T05:01:32.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:01:32.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673005 2024-09-15T05:01:32.299 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673005 2024-09-15T05:01:32.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-09-15T05:01:32.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:01:32.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673005 -lt 42949673005 2024-09-15T05:01:32.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:01:32.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509481 2024-09-15T05:01:32.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:01:32.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:01:32.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509481 2024-09-15T05:01:32.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:01:32.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509481 2024-09-15T05:01:32.671 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509481 2024-09-15T05:01:32.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-09-15T05:01:32.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:01:33.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509481 -lt 64424509481 2024-09-15T05:01:33.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:01:33.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345952 2024-09-15T05:01:33.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:01:33.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:01:33.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345952 2024-09-15T05:01:33.055 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:01:33.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345952 2024-09-15T05:01:33.056 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345952 2024-09-15T05:01:33.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2024-09-15T05:01:33.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:01:33.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345950 -lt 85899345952 2024-09-15T05:01:33.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:01:34.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:01:34.543 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:01:34.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345952 -lt 85899345952 2024-09-15T05:01:34.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:01:34.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182427 2024-09-15T05:01:34.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:01:34.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:01:34.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182427 2024-09-15T05:01:34.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:01:34.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182427 2024-09-15T05:01:34.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-09-15T05:01:34.955 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182427 2024-09-15T05:01:34.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:01:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182427 -lt 107374182427 2024-09-15T05:01:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:01:35.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018902 2024-09-15T05:01:35.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:01:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:01:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018902 2024-09-15T05:01:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:01:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018902 2024-09-15T05:01:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018902' 2024-09-15T05:01:35.380 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018902 2024-09-15T05:01:35.381 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:01:35.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018902 -lt 128849018902 2024-09-15T05:01:35.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:01:35.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:35.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:36.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:36.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:36.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:36.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:36.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:36.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:37.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:37.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:01:37.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:01:37.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:01:37.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:37.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:37.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:37.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:37.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:37.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:37.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:37.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:38.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:38.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-09-15T05:01:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-09-15T05:01:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:38.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:38.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:38.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:38.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:38.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:38.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:38.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:38.822 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:39.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:39.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:39.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:39.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:39.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:39.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:01:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:01:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:40.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:40.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:40.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:40.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:40.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:40.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:40.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:40.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:41.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:41.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:41.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:41.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:41.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:01:41.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:01:41.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:41.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:41.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:41.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:41.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:41.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:42.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:42.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:42.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:42.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:42.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:43.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-09-15T05:01:43.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-09-15T05:01:43.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:43.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:43.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:43.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:43.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:43.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:43.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:43.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:43.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:43.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:44.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:44.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:44.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:44.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:01:44.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:01:44.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:44.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:44.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:44.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:44.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:44.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:45.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:01:45.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:45.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:45.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:45.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:01:45.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:01:45.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:01:45.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:01:46.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-15T05:01:46.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-15T05:01:46.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:01:46.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:01:46.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:01:46.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:01:46.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:01:46.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:01:46.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:01:46.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:01:46.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:01:46.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:01:46.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:01:46.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:01:46.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:01:46.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:01:46.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:01:47.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:01:47.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:01:47.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:01:47.105 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:01:47.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-09-15T05:01:47.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:01:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:01:47.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-15T05:01:47.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-15T05:01:47.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:01:47.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:01:47.131 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:01:47.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:01:47.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:01:47.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:01:47.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:01:47.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:01:47.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:01:47.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:01:47.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:01:47.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-15T05:01:47.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-15T05:01:47.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:01:47.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:01:47.632 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:01:47.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-15T05:01:47.644 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:01:47.644 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:01:47.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:01:47.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:01:47.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:01:52.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:01:52.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:01:52.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:01:52.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:01:52.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:01:52.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:01:52.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:01:52.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:01:52.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:01:52.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:01:52.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:01:52.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:01:52.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:01:52.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:01:52.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:01:52.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:01:53.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:01:53.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:01:53.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:01:53.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:01:53.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:01:53.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:01:53.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:01:53.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:01:53.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:01:53.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:01:53.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:01:53.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:01:53.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:01:53.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:01:53.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:01:53.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:01:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:01:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:01:53.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:01:53.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:01:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:01:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:01:53.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:01:53.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:01:53.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:01:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:01:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:01:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:01:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:01:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:01:53.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:01:53.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:01:53.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:01:53.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:01:53.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:01:53.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:01:53.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:01:53.597 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:01:53.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:01:53.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:01:53.682 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:01:53.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:01:53.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-09-15T05:01:53.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:01:53.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:01:53.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:01:53.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:01:53.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:01:53.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:01:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:01:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:01:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:01:54.591 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:01:54.591 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:54.591 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:54.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:01:54.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:01:54.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:01:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-09-15T05:01:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:01:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:01:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:01:54.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:01:54.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:01:54.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=138431b0-2413-404d-8984-08312b0f15ab 2024-09-15T05:01:54.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 138431b0-2413-404d-8984-08312b0f15ab' 2024-09-15T05:01:54.626 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 138431b0-2413-404d-8984-08312b0f15ab 2024-09-15T05:01:54.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:01:54.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCauZmj4eoJRAAMgeBXLGdfdr/FTR0ClOOAw== 2024-09-15T05:01:54.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCauZmj4eoJRAAMgeBXLGdfdr/FTR0ClOOAw=="}' 2024-09-15T05:01:54.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 138431b0-2413-404d-8984-08312b0f15ab -i td/osd-recovery-stats/0/new.json 2024-09-15T05:01:54.968 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:01:54.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:01:54.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCauZmj4eoJRAAMgeBXLGdfdr/FTR0ClOOAw== --osd-uuid 138431b0-2413-404d-8984-08312b0f15ab 2024-09-15T05:01:55.003 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.002+0000 7feadb92f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:01:55.013 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.013+0000 7feadb92f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:01:55.014 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.014+0000 7feadb92f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:01:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.014+0000 7feadb92f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:01:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.014+0000 7feadb92f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:01:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.014+0000 7feadb92f440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:01:55.054 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:01:55.053+0000 7feadb92f440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:02:00.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:02:00.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:02:00.386 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:02:00.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:02:00.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:02:00.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:02:00.869 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:02:00.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:02:00.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:02:00.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:02:00.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:02:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:00.891+0000 7fa0e5f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:00.893 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:00.893+0000 7fa0e5f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:00.894 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:00.894+0000 7fa0e5f1d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:02:01.218 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:02:01.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:01.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:01.411 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:01.411+0000 7fa0e5f1d440 -1 Falling back to public interface 2024-09-15T05:02:01.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:02.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:02.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:02.596 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:02:02.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:02:02.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:02.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:02.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:03.893 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:03.892+0000 7fa0e5f1d440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:02:03.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:03.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:03.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:02:03.955 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:02:03.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:03.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:04.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:05.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:05.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:05.507 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:02:05.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:02:05.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:05.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:05.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:06.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:06.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:06.943 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:02:06.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:02:06.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:06.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:07.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:08.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:08.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:08.301 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:02:08.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:02:08.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:08.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:08.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:09.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:09.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:09.769 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:02:09.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:02:09.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:09.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:10.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:11.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:11.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:11.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:02:11.130 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:02:11.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:11.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:11.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:12.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:12.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:12.622 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:02:12.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:02:12.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:12.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:12.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:13.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:13.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:13.990 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:02:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:02:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:14.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:15.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:15.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:15.352 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:02:15.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:02:15.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:15.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:15.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:16.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:16.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:16.762 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:02:16.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:02:16.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:16.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:17.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:18.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:18.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:18.119 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:02:18.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:02:18.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:18.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:19.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:20.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:20.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:20.137 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:02:20.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:02:20.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:20.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:21.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:22.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:22.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:22.005 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:02:22.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:02:22.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:22.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:22.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:23.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:23.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:23.821 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:02:23.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:02:23.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:23.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:24.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:25.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:25.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:25.578 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:02:25.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:02:25.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:25.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:26.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:27.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:27.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:27.369 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:02:27.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:02:27.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:27.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:02:27.723 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/308498,v1:127.0.0.1:6803/308498] [v2:127.0.0.1:6804/308498,v1:127.0.0.1:6805/308498] exists,up 138431b0-2413-404d-8984-08312b0f15ab 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:02:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:02:27.728 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:02:27.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:02:27.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:02:27.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=efbcec0e-cd3a-47df-9553-902b9b154365 2024-09-15T05:02:27.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 efbcec0e-cd3a-47df-9553-902b9b154365' 2024-09-15T05:02:27.731 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 efbcec0e-cd3a-47df-9553-902b9b154365 2024-09-15T05:02:27.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:02:27.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjauZmlHYwLBAA+9qh1qoTiL758nMyh1cdCg== 2024-09-15T05:02:27.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjauZmlHYwLBAA+9qh1qoTiL758nMyh1cdCg=="}' 2024-09-15T05:02:27.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new efbcec0e-cd3a-47df-9553-902b9b154365 -i td/osd-recovery-stats/1/new.json 2024-09-15T05:02:28.251 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:02:28.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:02:28.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjauZmlHYwLBAA+9qh1qoTiL758nMyh1cdCg== --osd-uuid efbcec0e-cd3a-47df-9553-902b9b154365 2024-09-15T05:02:28.287 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.285+0000 7f9c00e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:28.310 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.308+0000 7f9c00e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:28.311 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.309+0000 7f9c00e87440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:28.311 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.309+0000 7f9c00e87440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:02:28.311 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.309+0000 7f9c00e87440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:02:28.311 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.309+0000 7f9c00e87440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:02:28.394 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:28.392+0000 7f9c00e87440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:02:34.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:02:34.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:02:34.199 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:02:34.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:02:34.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:02:34.687 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:02:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:02:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:02:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:02:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:02:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:02:34.710 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:34.709+0000 7f3336335440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:34.712 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:34.711+0000 7f3336335440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:34.713 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:34.712+0000 7f3336335440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:02:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:35.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:02:35.037 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:02:35.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:35.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:35.228 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:35.228+0000 7f3336335440 -1 Falling back to public interface 2024-09-15T05:02:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:36.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:36.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:36.519 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:02:36.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:02:36.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:36.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:36.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:37.717 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:02:37.716+0000 7f3336335440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:02:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:37.886 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:02:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:02:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:37.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:38.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:39.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:39.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:39.479 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:02:39.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:02:39.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:39.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:39.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:40.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:40.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:40.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:02:40.879 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:02:40.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:40.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:41.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:42.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:42.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:42.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:02:42.326 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:02:42.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:42.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:42.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:43.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:43.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:43.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:02:43.724 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:02:43.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:43.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:44.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:45.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:45.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:45.097 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:02:45.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:02:45.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:45.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:45.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:46.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:46.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:46.515 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:02:46.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:02:46.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:46.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:46.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:47.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:47.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:47.955 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:02:47.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:02:47.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:47.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:48.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:49.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:49.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:49.360 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:02:49.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:02:49.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:49.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:49.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:50.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:50.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:50.860 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:02:50.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:02:50.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:50.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:51.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:52.481 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:02:52.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:02:52.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:52.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:53.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:54.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:54.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:54.259 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:02:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:02:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:55.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:56.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:56.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:56.278 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:02:56.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:02:56.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:56.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:57.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:58.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:58.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:58.246 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:02:58.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:02:58.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:58.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:02:58.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:02:59.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:02:59.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:02:59.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:02:59.931 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:02:59.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:02:59.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:03:00.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:01.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:01.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:01.556 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:03:01.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:03:01.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:01.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:03:02.107 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/310252,v1:127.0.0.1:6811/310252] [v2:127.0.0.1:6812/310252,v1:127.0.0.1:6813/310252] exists,up efbcec0e-cd3a-47df-9553-902b9b154365 2024-09-15T05:03:02.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:03:02.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:03:02.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:03:02.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:03:02.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:03:02.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:03:02.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=305e4be5-50a5-45c1-b606-b27c3507e698 2024-09-15T05:03:02.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 305e4be5-50a5-45c1-b606-b27c3507e698' 2024-09-15T05:03:02.115 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 305e4be5-50a5-45c1-b606-b27c3507e698 2024-09-15T05:03:02.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:03:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGauZmJiFzBxAAk1pYNdxVl420iGFSDhfXMA== 2024-09-15T05:03:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGauZmJiFzBxAAk1pYNdxVl420iGFSDhfXMA=="}' 2024-09-15T05:03:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 305e4be5-50a5-45c1-b606-b27c3507e698 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:03:02.601 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:03:02.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:03:02.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGauZmJiFzBxAAk1pYNdxVl420iGFSDhfXMA== --osd-uuid 305e4be5-50a5-45c1-b606-b27c3507e698 2024-09-15T05:03:02.637 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.636+0000 7fb6acc12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:02.639 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.638+0000 7fb6acc12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:02.640 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.639+0000 7fb6acc12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:02.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.640+0000 7fb6acc12440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:03:02.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.640+0000 7fb6acc12440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:03:02.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.640+0000 7fb6acc12440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:03:02.717 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:02.716+0000 7fb6acc12440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:03:08.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:03:08.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:03:08.584 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:03:08.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:03:08.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:03:09.070 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:03:09.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:03:09.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:03:09.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:03:09.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:03:09.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:03:09.093 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:09.092+0000 7f0a7ea8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:09.095 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:09.095+0000 7f0a7ea8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:09.096 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:09.096+0000 7f0a7ea8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:09.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:03:09.420 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:03:09.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:09.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:09.617 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:09.616+0000 7f0a7ea8b440 -1 Falling back to public interface 2024-09-15T05:03:09.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:10.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:10.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:10.775 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:03:10.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:03:10.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:10.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:11.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:12.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:12.088+0000 7f0a7ea8b440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:03:12.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:12.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:12.122 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:03:12.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:03:12.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:12.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:12.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:13.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:13.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:13.533 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:03:13.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:03:13.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:13.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:14.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:15.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:15.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:15.029 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:03:15.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:03:15.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:15.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:15.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:16.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:16.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:16.410 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:03:16.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:03:16.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:16.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:16.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:17.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:17.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:17.826 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:03:17.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:03:17.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:17.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:18.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:19.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:19.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:19.197 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:03:19.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:03:19.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:19.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:19.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:20.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:20.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:20.594 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:03:20.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:03:20.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:20.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:20.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:21.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:21.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:21.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:03:21.955 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:03:21.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:21.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:22.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:23.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:23.339 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:03:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:03:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:23.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:24.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:24.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:24.959 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:03:24.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:03:24.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:24.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:26.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:27.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:27.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:27.067 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:03:27.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:03:27.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:27.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:27.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:28.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:28.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:28.541 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:03:28.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:03:28.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:28.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:29.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:30.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:30.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:30.394 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:03:30.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:03:30.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:30.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:31.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:32.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:32.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:32.258 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:03:32.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:03:32.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:32.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:33.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:34.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:34.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:34.345 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:03:34.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:03:34.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:34.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:34.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:35.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:35.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:35.786 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:03:35.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:03:35.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:35.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:03:36.153 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/312007,v1:127.0.0.1:6819/312007] [v2:127.0.0.1:6820/312007,v1:127.0.0.1:6821/312007] exists,up 305e4be5-50a5-45c1-b606-b27c3507e698 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:03:36.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:03:36.158 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:03:36.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f95102fc-37cf-4b4a-a326-b9f04ec051c7 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 f95102fc-37cf-4b4a-a326-b9f04ec051c7' 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 f95102fc-37cf-4b4a-a326-b9f04ec051c7 2024-09-15T05:03:36.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:03:36.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCoauZmelYiChAAy2TzClc3ZKnVOm+db+G5pA== 2024-09-15T05:03:36.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCoauZmelYiChAAy2TzClc3ZKnVOm+db+G5pA=="}' 2024-09-15T05:03:36.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f95102fc-37cf-4b4a-a326-b9f04ec051c7 -i td/osd-recovery-stats/3/new.json 2024-09-15T05:03:36.649 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:03:36.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T05:03:36.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCoauZmelYiChAAy2TzClc3ZKnVOm+db+G5pA== --osd-uuid f95102fc-37cf-4b4a-a326-b9f04ec051c7 2024-09-15T05:03:36.685 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.685+0000 7fba8ca0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:36.687 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.687+0000 7fba8ca0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:36.688 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.688+0000 7fba8ca0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.688+0000 7fba8ca0c440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:03:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.688+0000 7fba8ca0c440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:03:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.688+0000 7fba8ca0c440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:03:36.765 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:36.764+0000 7fba8ca0c440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T05:03:42.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T05:03:42.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:03:42.338 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:03:42.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:03:42.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:03:42.825 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:03:42.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:03:42.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:03:42.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:03:42.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:03:42.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:03:42.847 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:42.847+0000 7fbb8fb66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:42.872 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:42.871+0000 7fbb8fb66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:42.872 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:42.872+0000 7fbb8fb66440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:03:43.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:03:43.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:03:43.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:03:43.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:43.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:43.398 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:43.397+0000 7fbb8fb66440 -1 Falling back to public interface 2024-09-15T05:03:43.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:44.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:44.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:44.608 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:03:44.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:03:44.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:44.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:45.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:03:45.883+0000 7fbb8fb66440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:03:45.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:45.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:45.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:03:45.989 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:03:45.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:45.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:46.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:47.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:47.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:47.503 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:03:47.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:03:47.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:47.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:47.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:48.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:48.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:48.985 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:03:48.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:03:48.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:48.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:49.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:50.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:50.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:50.403 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:03:50.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:03:50.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:50.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:50.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:51.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:51.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:51.889 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:03:51.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:03:51.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:51.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:53.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:53.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:53.248 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:03:53.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:03:53.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:53.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:53.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:54.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:54.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:54.670 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:03:54.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:03:54.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:54.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:55.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:56.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:56.187 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:03:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:03:56.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:56.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:56.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:57.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:57.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:57.724 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:03:57.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:03:57.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:57.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:03:58.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:03:59.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:03:59.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:03:59.163 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:03:59.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:03:59.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:03:59.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:00.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:01.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:01.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:01.007 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:04:01.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:04:01.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:01.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:01.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:02.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:02.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:02.894 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:04:02.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:04:02.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:02.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:03.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:04.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:04.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:04.820 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:04:04.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:04:04.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:04.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:05.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:06.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:06.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:06.554 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:04:06.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:04:06.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:06.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:07.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:08.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:08.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:08.376 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:04:08.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:04:08.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:08.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:09.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:04:10.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:04:10.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:04:10.187 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:04:10.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:04:10.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:04:10.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:04:10.567 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/313761,v1:127.0.0.1:6827/313761] [v2:127.0.0.1:6828/313761,v1:127.0.0.1:6829/313761] exists,up f95102fc-37cf-4b4a-a326-b9f04ec051c7 2024-09-15T05:04:10.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:04:10.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:04:10.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:04:10.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T05:04:11.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-09-15T05:04:11.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-09-15T05:04:11.944 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:04:11.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:04:12.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-09-15T05:04:13.976 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T05:04:13.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-15T05:04:13.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:04:13.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:04:13.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:04:13.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:04:13.987 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:04:13.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:04:13.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:04:13.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:04:13.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:04:14.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:04:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:04:14.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:04:14.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T05:04:14.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T05:04:14.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T05:04:14.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:04:14.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:04:14.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-15T05:04:14.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-15T05:04:14.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672980' 2024-09-15T05:04:14.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:04:14.687 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-15T05:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-15T05:04:14.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672980 2-64424509452' 2024-09-15T05:04:14.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:04:14.814 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:04:14.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345932 2024-09-15T05:04:14.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345932 2024-09-15T05:04:14.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672980 2-64424509452 3-85899345932' 2024-09-15T05:04:14.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:04:14.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T05:04:14.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:04:14.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:04:14.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T05:04:14.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:04:14.919 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836509 2024-09-15T05:04:14.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T05:04:14.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T05:04:14.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:04:15.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-15T05:04:15.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:04:16.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:04:16.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:04:16.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T05:04:16.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:04:16.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-15T05:04:16.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:04:16.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:04:16.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-15T05:04:16.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:04:16.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-15T05:04:16.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T05:04:16.669 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672980 2024-09-15T05:04:16.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:04:17.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-15T05:04:17.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:04:17.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-15T05:04:17.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:04:17.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:04:17.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-15T05:04:17.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:04:17.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-15T05:04:17.021 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509452 2024-09-15T05:04:17.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T05:04:17.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:04:17.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-15T05:04:17.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:04:17.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345932 2024-09-15T05:04:17.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:04:17.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:04:17.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345932 2024-09-15T05:04:17.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:04:17.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345932 2024-09-15T05:04:17.372 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345932 2024-09-15T05:04:17.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2024-09-15T05:04:17.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:04:17.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345933 -lt 85899345932 2024-09-15T05:04:17.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:04:17.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:04:17.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:04:18.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:04:18.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:04:18.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:04:18.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:04:18.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:04:18.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:04:19.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:04:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:04:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:04:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-09-15T05:04:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-09-15T05:04:19.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-09-15T05:04:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-09-15T05:04:19.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-09-15T05:04:19.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-09-15T05:04:19.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:19.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-09-15T05:04:20.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-09-15T05:04:20.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-09-15T05:04:20.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-09-15T05:04:20.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-09-15T05:04:20.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-09-15T05:04:20.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:20.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-09-15T05:04:21.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:21.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-09-15T05:04:21.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:21.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-09-15T05:04:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:21.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-09-15T05:04:21.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:21.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-09-15T05:04:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-09-15T05:04:22.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-09-15T05:04:22.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-09-15T05:04:22.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-09-15T05:04:22.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-09-15T05:04:22.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-09-15T05:04:22.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:22.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-09-15T05:04:23.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:23.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-09-15T05:04:23.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:23.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-09-15T05:04:23.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:23.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-09-15T05:04:23.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:23.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-09-15T05:04:23.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:23.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-09-15T05:04:24.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:24.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-09-15T05:04:24.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:24.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-09-15T05:04:24.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:24.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-09-15T05:04:24.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:24.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-09-15T05:04:24.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:24.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-09-15T05:04:25.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-09-15T05:04:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-09-15T05:04:25.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-09-15T05:04:25.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-09-15T05:04:25.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-09-15T05:04:25.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:25.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-09-15T05:04:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-09-15T05:04:26.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:26.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-09-15T05:04:26.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:26.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-09-15T05:04:26.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:26.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-09-15T05:04:26.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:26.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-09-15T05:04:27.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-09-15T05:04:27.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-09-15T05:04:27.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-09-15T05:04:27.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-09-15T05:04:27.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-09-15T05:04:27.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:27.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-09-15T05:04:28.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:28.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-09-15T05:04:28.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:28.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-09-15T05:04:28.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:28.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-09-15T05:04:28.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:28.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-09-15T05:04:28.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:28.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-09-15T05:04:29.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:29.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-09-15T05:04:29.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:29.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-09-15T05:04:29.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:29.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-09-15T05:04:29.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:29.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-09-15T05:04:29.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:29.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-09-15T05:04:30.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-09-15T05:04:30.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-09-15T05:04:30.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-09-15T05:04:30.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-09-15T05:04:30.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-09-15T05:04:30.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:30.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-09-15T05:04:31.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:31.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-09-15T05:04:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-09-15T05:04:31.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:31.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-09-15T05:04:31.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:31.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-09-15T05:04:31.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:31.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-09-15T05:04:32.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:32.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-09-15T05:04:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:32.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-09-15T05:04:32.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:32.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-09-15T05:04:32.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:32.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-09-15T05:04:32.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:32.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-09-15T05:04:33.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:33.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-09-15T05:04:33.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:33.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-09-15T05:04:33.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:33.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-09-15T05:04:33.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:33.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-09-15T05:04:33.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:33.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-09-15T05:04:34.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:34.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-09-15T05:04:34.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:34.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-09-15T05:04:34.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:34.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-09-15T05:04:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:34.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-09-15T05:04:34.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:34.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-09-15T05:04:35.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-09-15T05:04:35.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-09-15T05:04:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-09-15T05:04:35.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-09-15T05:04:35.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-09-15T05:04:35.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:35.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-09-15T05:04:36.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:36.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-09-15T05:04:36.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:36.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-09-15T05:04:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:36.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-09-15T05:04:36.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:36.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-09-15T05:04:36.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:36.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-09-15T05:04:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-09-15T05:04:37.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-09-15T05:04:37.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-09-15T05:04:37.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-09-15T05:04:37.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-09-15T05:04:37.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:37.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-09-15T05:04:38.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:38.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-09-15T05:04:38.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:38.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-09-15T05:04:38.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:38.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-09-15T05:04:38.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:38.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-09-15T05:04:38.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:38.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-09-15T05:04:39.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-09-15T05:04:39.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-09-15T05:04:39.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-09-15T05:04:39.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-09-15T05:04:39.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-09-15T05:04:39.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:39.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-09-15T05:04:40.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:40.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-09-15T05:04:40.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:40.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-09-15T05:04:40.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:40.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-09-15T05:04:40.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:40.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-09-15T05:04:40.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:40.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-09-15T05:04:41.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:41.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-09-15T05:04:41.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:41.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-09-15T05:04:41.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:41.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-09-15T05:04:41.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:41.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-09-15T05:04:41.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:41.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-09-15T05:04:42.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-09-15T05:04:42.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-09-15T05:04:42.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-09-15T05:04:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-09-15T05:04:42.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-09-15T05:04:42.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:42.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-09-15T05:04:43.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:43.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-09-15T05:04:43.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:43.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-09-15T05:04:43.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:43.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-09-15T05:04:43.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:43.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-09-15T05:04:43.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:43.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-09-15T05:04:44.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:44.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-09-15T05:04:44.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:44.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-09-15T05:04:44.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:44.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-09-15T05:04:44.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:44.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-09-15T05:04:44.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:44.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-09-15T05:04:45.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-09-15T05:04:45.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-09-15T05:04:45.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-09-15T05:04:45.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-09-15T05:04:45.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-09-15T05:04:45.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:45.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-09-15T05:04:46.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:46.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-09-15T05:04:46.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:46.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-09-15T05:04:46.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:46.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-09-15T05:04:46.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:46.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-09-15T05:04:46.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:46.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-09-15T05:04:47.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:47.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-09-15T05:04:47.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:47.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-09-15T05:04:47.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:47.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-09-15T05:04:47.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:47.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-09-15T05:04:47.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:47.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-09-15T05:04:48.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-09-15T05:04:48.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-09-15T05:04:48.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-09-15T05:04:48.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-09-15T05:04:48.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-09-15T05:04:48.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:48.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-09-15T05:04:49.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:49.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-09-15T05:04:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:49.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-09-15T05:04:49.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:49.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-09-15T05:04:49.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:49.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-09-15T05:04:49.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:49.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-09-15T05:04:50.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:50.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-09-15T05:04:50.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:50.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-09-15T05:04:50.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:50.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-09-15T05:04:50.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:50.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-09-15T05:04:50.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:50.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-09-15T05:04:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-09-15T05:04:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-09-15T05:04:51.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-09-15T05:04:51.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-09-15T05:04:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-09-15T05:04:51.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:51.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-09-15T05:04:52.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:52.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-09-15T05:04:52.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:52.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-09-15T05:04:52.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:52.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-09-15T05:04:52.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:52.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-09-15T05:04:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:52.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-09-15T05:04:53.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:53.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-09-15T05:04:53.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:53.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-09-15T05:04:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-09-15T05:04:53.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:53.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-09-15T05:04:53.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:53.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-09-15T05:04:54.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:54.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-09-15T05:04:54.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:54.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-09-15T05:04:54.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:54.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-09-15T05:04:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:54.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-09-15T05:04:54.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:54.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-09-15T05:04:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-09-15T05:04:55.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-09-15T05:04:55.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-09-15T05:04:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-09-15T05:04:55.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-09-15T05:04:55.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:55.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-09-15T05:04:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-09-15T05:04:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-09-15T05:04:56.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-15T05:04:56.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:04:56.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:04:56.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:04:56.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:04:56.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-09-15T05:04:56.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-09-15T05:04:56.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:04:56.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:04:56.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:04:56.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:04:57.103 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:04:57.104 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:04:57.104 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:04:57.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T05:04:57.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-15T05:04:57.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T05:04:57.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-09-15T05:04:57.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-09-15T05:04:58.537 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:04:58.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-09-15T05:04:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 308498 2024-09-15T05:04:58.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-09-15T05:04:59.208 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:04:59.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-09-15T05:04:59.798 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:04:59.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-09-15T05:04:59.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-09-15T05:05:00.361 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T05:05:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-09-15T05:05:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-09-15T05:05:01.077 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:05:01.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-09-15T05:05:01.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:05:01.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:05:01.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:05:01.088 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:05:01.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T05:05:01.556 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:05:01.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-09-15T05:05:03.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-09-15T05:05:03.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:05:03.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:05:03.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:05:03.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:05:03.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:05:03.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:05:03.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:05:03.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:05:03.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:05:03.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:05:04.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:05:04.129 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-15T05:05:04.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T05:05:04.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T05:05:04.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T05:05:04.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:05:04.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:05:04.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672992 2024-09-15T05:05:04.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672992 2024-09-15T05:05:04.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992' 2024-09-15T05:05:04.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:05:04.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:05:04.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509464 2024-09-15T05:05:04.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509464 2024-09-15T05:05:04.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992 2-64424509464' 2024-09-15T05:05:04.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:05:04.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345944 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345944 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949672992 2-64424509464 3-85899345944' 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672992 2024-09-15T05:05:04.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:05:04.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:05:04.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672992 2024-09-15T05:05:04.464 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:05:04.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672992 2024-09-15T05:05:04.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-15T05:05:04.465 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672992 2024-09-15T05:05:04.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:05:04.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672990 -lt 42949672992 2024-09-15T05:05:04.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:05:05.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:05:05.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:05:06.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-15T05:05:06.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:05:06.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509464 2024-09-15T05:05:06.316 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:05:06.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:05:06.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509464 2024-09-15T05:05:06.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:05:06.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509464 2024-09-15T05:05:06.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509464' 2024-09-15T05:05:06.319 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509464 2024-09-15T05:05:06.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:05:06.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509464 2024-09-15T05:05:06.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:05:06.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345944 2024-09-15T05:05:06.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:05:06.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:05:06.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345944 2024-09-15T05:05:06.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:05:06.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345944 2024-09-15T05:05:06.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345944' 2024-09-15T05:05:06.763 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345944 2024-09-15T05:05:06.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:05:07.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345945 -lt 85899345944 2024-09-15T05:05:07.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:05:07.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:07.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:07.820 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:08.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:08.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:08.248 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:08.248 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:08.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:08.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:09.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:09.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:09.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:09.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:09.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:09.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:05:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:05:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:10.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:10.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:10.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:10.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:11.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:11.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:05:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:05:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:11.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:11.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:11.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:11.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:11.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:11.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:11.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:11.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:12.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:12.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:12.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:12.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:12.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:12.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:12.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:13.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:05:13.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:05:13.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:13.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:13.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:13.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:13.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:13.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:13.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:13.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:14.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:14.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:14.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:14.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17 2024-09-15T05:05:14.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17 '!=' null 2024-09-15T05:05:14.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:14.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:14.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:15.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:15.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:15.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:15.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:15.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:15.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:15.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:15.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:16.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-15T05:05:16.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-15T05:05:16.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:16.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:16.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:16.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:16.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:16.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:16.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:16.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:16.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:16.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:16.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:16.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:05:16.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:16.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:16.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:05:17.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:05:17.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:05:17.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:05:17.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-15T05:05:17.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-15T05:05:17.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:05:17.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:05:17.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:05:17.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:05:17.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:05:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:05:18.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:05:18.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:05:18.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:05:18.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:05:18.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:05:18.674 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.675 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:05:18.675 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:05:18.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-09-15T05:05:18.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:05:18.691 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:05:18.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-15T05:05:18.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:05:18.698 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:05:18.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:05:18.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-09-15T05:05:18.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:05:18.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:05:18.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:05:18.718 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:05:18.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:05:18.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:05:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:05:19.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-09-15T05:05:19.217 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:05:19.217 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:05:19.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:05:19.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:05:19.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:05:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:05:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:05:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:05:24.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:05:24.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:05:24.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:05:24.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:05:24.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:05:24.538 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:05:24.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:05:24.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:05:24.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:05:24.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:05:24.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:05:24.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:05:24.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:05:24.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:05:24.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:05:24.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:05:24.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:05:24.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:24.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:24.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:05:24.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:05:24.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:05:24.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:05:24.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:05:24.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:05:24.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:05:24.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:05:24.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:05:24.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:05:24.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:05:24.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:05:24.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:05:24.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:05:24.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:05:24.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:05:24.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:05:24.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:05:24.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:24.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:24.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:05:24.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:05:24.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:05:24.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:05:24.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:05:24.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:24.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:24.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-09-15T05:05:24.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:05:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:05:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:05:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:05:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:05:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:05:24.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:05:25.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:05:25.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:05:25.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:05:25.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:05:25.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:05:25.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:25.268 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:25.269 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:05:25.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:05:25.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:05:25.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-09-15T05:05:25.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:05:25.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:05:25.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:05:25.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:05:25.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:05:25.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:05:26.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:26.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:05:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:05:26.213 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-09-15T05:05:26.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-09-15T05:05:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:05:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:05:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:05:26.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:05:26.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1b11038f-c338-42a9-9887-52710ad0e257 2024-09-15T05:05:26.222 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 1b11038f-c338-42a9-9887-52710ad0e257 2024-09-15T05:05:26.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1b11038f-c338-42a9-9887-52710ad0e257' 2024-09-15T05:05:26.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:05:26.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWa+ZmAmDTDRAAFGaIzHL9vLHBxfDnRCuxkw== 2024-09-15T05:05:26.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWa+ZmAmDTDRAAFGaIzHL9vLHBxfDnRCuxkw=="}' 2024-09-15T05:05:26.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1b11038f-c338-42a9-9887-52710ad0e257 -i td/osd-recovery-stats/0/new.json 2024-09-15T05:05:26.523 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:05:26.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:05:26.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWa+ZmAmDTDRAAFGaIzHL9vLHBxfDnRCuxkw== --osd-uuid 1b11038f-c338-42a9-9887-52710ad0e257 2024-09-15T05:05:26.558 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.557+0000 7f20b1887440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:26.570 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.569+0000 7f20b1887440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:26.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.570+0000 7f20b1887440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:26.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.571+0000 7f20b1887440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:05:26.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.571+0000 7f20b1887440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:05:26.572 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.571+0000 7f20b1887440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:05:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:26.618+0000 7f20b1887440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:05:31.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:05:31.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:05:31.990 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:05:31.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:05:31.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:05:32.472 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:05:32.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:05:32.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:05:32.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:05:32.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:05:32.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:05:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:32.494+0000 7f1e78584440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:32.496+0000 7f1e78584440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:32.497 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:32.497+0000 7f1e78584440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:05:32.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:05:32.822 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:05:32.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:32.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:33.012 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:33.012+0000 7f1e78584440 -1 Falling back to public interface 2024-09-15T05:05:33.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:34.188 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:05:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:05:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:34.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:05:35.517+0000 7f1e78584440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:05:35.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:35.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:35.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:05:35.544 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:05:35.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:35.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:35.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:36.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:36.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:36.926 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:05:36.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:05:36.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:36.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:37.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:38.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:38.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:38.407 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:05:38.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:05:38.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:38.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:38.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:39.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:39.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:39.796 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:05:39.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:05:39.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:39.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:40.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:41.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:41.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:41.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:05:41.152 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:05:41.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:41.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:42.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:42.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:42.504 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:05:42.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:05:42.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:42.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:42.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:43.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:43.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:43.868 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:05:43.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:05:43.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:43.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:44.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:45.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:45.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:45.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:05:45.391 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:05:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:45.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:46.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:46.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:46.744 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:05:46.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:05:46.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:46.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:47.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:48.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:48.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:48.127 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:05:48.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:05:48.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:48.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:48.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:49.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:49.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:49.947 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:05:49.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:05:49.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:49.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:51.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:52.158 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:05:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:05:52.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:52.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:52.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:53.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:53.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:53.902 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:05:53.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:05:53.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:53.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:54.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:55.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:55.671 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:05:55.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:05:55.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:55.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:56.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:57.696 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:05:57.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:05:57.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:57.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:58.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:05:59.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:05:59.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:05:59.209 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:05:59.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:05:59.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:05:59.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:05:59.566 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/321465,v1:127.0.0.1:6803/321465] [v2:127.0.0.1:6804/321465,v1:127.0.0.1:6805/321465] exists,up 1b11038f-c338-42a9-9887-52710ad0e257 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:05:59.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:05:59.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:05:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dbe270c0-a63a-459c-9ca9-f16fca156e74 2024-09-15T05:05:59.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dbe270c0-a63a-459c-9ca9-f16fca156e74' 2024-09-15T05:05:59.574 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 dbe270c0-a63a-459c-9ca9-f16fca156e74 2024-09-15T05:05:59.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:05:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3a+ZmsALFIhAAeWyW8XNfqQZo8++A1iDidw== 2024-09-15T05:05:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3a+ZmsALFIhAAeWyW8XNfqQZo8++A1iDidw=="}' 2024-09-15T05:05:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dbe270c0-a63a-459c-9ca9-f16fca156e74 -i td/osd-recovery-stats/1/new.json 2024-09-15T05:06:00.086 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:06:00.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:06:00.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3a+ZmsALFIhAAeWyW8XNfqQZo8++A1iDidw== --osd-uuid dbe270c0-a63a-459c-9ca9-f16fca156e74 2024-09-15T05:06:00.122 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.121+0000 7fed0102f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:00.123 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.123+0000 7fed0102f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.124+0000 7fed0102f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.124+0000 7fed0102f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:06:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.124+0000 7fed0102f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:06:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.124+0000 7fed0102f440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:06:00.201 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:00.200+0000 7fed0102f440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:06:06.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:06:06.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:06:06.076 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:06:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:06:06.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:06:06.561 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:06:06.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:06:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:06:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:06:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:06:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:06:06.581 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:06.580+0000 7fd30057d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:06.609 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:06.608+0000 7fd30057d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:06.610 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:06.609+0000 7fd30057d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:06:06.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:06.907 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:06:06.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:06:06.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:06.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:07.143 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:07.143+0000 7fd30057d440 -1 Falling back to public interface 2024-09-15T05:06:07.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:08.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:08.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:08.266 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:06:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:06:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:09.615 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:09.614+0000 7fd30057d440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:06:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:09.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:09.694 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:06:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:06:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:10.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:11.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:11.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:11.346 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:06:11.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:06:11.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:11.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:11.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:12.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:12.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:12.768 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:06:12.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:06:12.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:12.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:13.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:14.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:14.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:14.139 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:06:14.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:06:14.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:14.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:14.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:15.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:15.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:15.587 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:06:15.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:06:15.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:15.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:16.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:17.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:17.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:17.019 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:06:17.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:06:17.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:17.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:17.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:18.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:18.488 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:06:18.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:18.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:06:18.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:18.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:18.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:19.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:19.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:19.899 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:06:19.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:06:19.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:19.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:20.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:21.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:21.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:21.255 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:06:21.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:06:21.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:21.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:21.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:22.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:22.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:22.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:06:22.776 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:06:22.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:22.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:23.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:24.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:24.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:24.534 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:06:24.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:06:24.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:24.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:25.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:26.291 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:06:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:06:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:26.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:27.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:28.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:28.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:28.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:06:28.419 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:06:28.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:28.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:29.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:30.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:30.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:30.179 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:06:30.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:06:30.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:30.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:30.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:31.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:31.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:31.942 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:06:31.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:06:31.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:31.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:32.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:33.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:33.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:33.736 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:06:33.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:06:33.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:33.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:06:34.106 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/323220,v1:127.0.0.1:6811/323220] [v2:127.0.0.1:6812/323220,v1:127.0.0.1:6813/323220] exists,up dbe270c0-a63a-459c-9ca9-f16fca156e74 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:06:34.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:06:34.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:06:34.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6aa9b6f7-fd0a-4f58-baba-a317dc5135c4 2024-09-15T05:06:34.113 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 6aa9b6f7-fd0a-4f58-baba-a317dc5135c4 2024-09-15T05:06:34.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6aa9b6f7-fd0a-4f58-baba-a317dc5135c4' 2024-09-15T05:06:34.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:06:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaa+ZmwhlfBxAAGyYYReIbAolyrWZz41sbhw== 2024-09-15T05:06:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaa+ZmwhlfBxAAGyYYReIbAolyrWZz41sbhw=="}' 2024-09-15T05:06:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6aa9b6f7-fd0a-4f58-baba-a317dc5135c4 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:06:34.622 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:06:34.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:06:34.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaa+ZmwhlfBxAAGyYYReIbAolyrWZz41sbhw== --osd-uuid 6aa9b6f7-fd0a-4f58-baba-a317dc5135c4 2024-09-15T05:06:34.658 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.657+0000 7f8b84f2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:34.660 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.659+0000 7f8b84f2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:34.661 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.660+0000 7f8b84f2f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:34.661 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.661+0000 7f8b84f2f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:06:34.661 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.661+0000 7f8b84f2f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:06:34.662 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.661+0000 7f8b84f2f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:06:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:34.737+0000 7f8b84f2f440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:06:40.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:06:40.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:06:40.573 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:06:40.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:06:40.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:06:41.055 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:06:41.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:06:41.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:06:41.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:06:41.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:06:41.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:06:41.077 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:41.076+0000 7fc624069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:41.079 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:41.079+0000 7fc624069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:41.080 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:41.080+0000 7fc624069440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:41.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:06:41.405 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:06:41.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:41.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:41.598 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:41.598+0000 7fc624069440 -1 Falling back to public interface 2024-09-15T05:06:41.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:42.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:42.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:42.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:06:42.753 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:06:42.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:42.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:43.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:44.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:44.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:44.105 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:06:44.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:06:44.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:44.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:44.122 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:06:44.122+0000 7fc624069440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:06:44.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:45.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:45.659 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:06:45.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:06:45.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:45.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:46.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:47.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:47.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:47.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:06:47.140 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:06:47.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:47.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:47.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:48.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:48.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:48.539 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:06:48.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:06:48.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:48.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:48.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:49.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:49.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:49.999 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:06:49.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:06:49.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:49.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:50.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:51.362 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:06:51.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:06:51.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:51.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:51.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:52.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:52.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:52.758 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:06:52.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:06:52.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:52.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:53.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:54.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:54.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:54.148 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:06:54.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:06:54.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:54.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:54.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:55.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:55.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:55.532 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:06:55.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:06:55.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:55.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:55.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:56.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:56.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:56.995 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:06:56.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:06:56.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:56.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:57.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:06:58.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:06:58.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:06:58.529 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:06:58.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:06:58.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:06:58.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:06:59.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:00.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:00.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:00.268 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:07:00.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:07:00.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:00.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:07:01.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:02.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:02.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:02.453 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:07:02.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:07:02.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:02.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:07:03.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:04.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:04.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:04.194 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:07:04.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:07:04.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:04.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:07:05.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:06.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:06.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:06.045 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:07:06.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:07:06.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:06.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:07:07.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:08.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:08.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:08.261 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:07:08.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:07:08.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:08.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:07:08.690 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/324974,v1:127.0.0.1:6819/324974] [v2:127.0.0.1:6820/324974,v1:127.0.0.1:6821/324974] exists,up 6aa9b6f7-fd0a-4f58-baba-a317dc5135c4 2024-09-15T05:07:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:07:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:07:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:07:08.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T05:07:08.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:07:08.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:07:08.697 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 bb01f11d-f846-45db-bbc4-b37ad69a186e 2024-09-15T05:07:08.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb01f11d-f846-45db-bbc4-b37ad69a186e 2024-09-15T05:07:08.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bb01f11d-f846-45db-bbc4-b37ad69a186e' 2024-09-15T05:07:08.698 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:07:08.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8a+ZmNfIuKhAApTBQwC/c+H/Dspz+dp+jHQ== 2024-09-15T05:07:08.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8a+ZmNfIuKhAApTBQwC/c+H/Dspz+dp+jHQ=="}' 2024-09-15T05:07:08.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb01f11d-f846-45db-bbc4-b37ad69a186e -i td/osd-recovery-stats/3/new.json 2024-09-15T05:07:09.156 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:07:09.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T05:07:09.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8a+ZmNfIuKhAApTBQwC/c+H/Dspz+dp+jHQ== --osd-uuid bb01f11d-f846-45db-bbc4-b37ad69a186e 2024-09-15T05:07:09.192 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.191+0000 7f6ba9c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:09.243 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.242+0000 7f6ba9c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:09.244 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.243+0000 7f6ba9c0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:09.244 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.243+0000 7f6ba9c0d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:07:09.244 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.244+0000 7f6ba9c0d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:07:09.244 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.244+0000 7f6ba9c0d440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:07:09.291 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:09.291+0000 7f6ba9c0d440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T05:07:15.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T05:07:15.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:07:15.404 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:07:15.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:07:15.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:07:15.907 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:07:15.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:07:15.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:07:15.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:07:15.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:07:15.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:07:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:15.928+0000 7ff3fe60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:15.932 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:15.930+0000 7ff3fe60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:15.933 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:15.931+0000 7ff3fe60c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:16.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:07:16.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:07:16.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:07:16.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:16.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:16.448 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:16.447+0000 7ff3fe60c440 -1 Falling back to public interface 2024-09-15T05:07:16.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:17.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:17.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:17.604 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:07:17.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:07:17.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:17.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:17.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:18.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:18.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:18.962 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:07:18.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:07:18.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:18.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:18.994 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:18.993+0000 7ff3fe60c440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:07:19.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:20.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:20.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:20.329 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:07:20.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:07:20.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:20.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:20.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:21.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:21.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:21.728 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:07:21.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:07:21.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:21.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:22.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:23.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:23.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:23.107 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:07:23.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:07:23.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:23.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:23.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:24.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:24.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:24.494 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:07:24.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:07:24.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:24.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:24.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:25.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:25.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:25.913 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:07:25.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:07:25.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:25.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:26.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:27.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:27.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:27.300 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:07:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:07:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:27.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:28.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:28.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:28.785 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:07:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:07:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:29.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:30.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:30.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:30.214 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:07:30.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:07:30.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:30.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:30.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:31.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:31.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:31.733 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:07:31.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:07:31.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:31.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:33.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:34.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:34.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:34.025 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:07:34.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:07:34.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:34.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:34.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:35.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:35.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:35.867 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:07:35.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:07:35.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:35.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:37.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:37.636 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:07:37.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:37.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:07:37.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:37.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:39.472 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:07:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:07:39.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:39.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:40.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:41.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:41.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:41.639 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:07:41.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:07:41.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:41.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:42.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:43.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:43.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:43.242 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:07:43.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:07:43.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:43.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:07:43.620 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/326728,v1:127.0.0.1:6827/326728] [v2:127.0.0.1:6828/326728,v1:127.0.0.1:6829/326728] exists,up bb01f11d-f846-45db-bbc4-b37ad69a186e 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:07:43.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:07:43.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:07:43.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=492f11f2-9231-4583-b663-cec5ed01049e 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 492f11f2-9231-4583-b663-cec5ed01049e' 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 492f11f2-9231-4583-b663-cec5ed01049e 2024-09-15T05:07:43.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:07:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfa+ZmS9oEJhAA/Uh9fgm/0JbEYDr/Hbs6tg== 2024-09-15T05:07:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfa+ZmS9oEJhAA/Uh9fgm/0JbEYDr/Hbs6tg=="}' 2024-09-15T05:07:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 492f11f2-9231-4583-b663-cec5ed01049e -i td/osd-recovery-stats/4/new.json 2024-09-15T05:07:44.150 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:07:44.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-15T05:07:44.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfa+ZmS9oEJhAA/Uh9fgm/0JbEYDr/Hbs6tg== --osd-uuid 492f11f2-9231-4583-b663-cec5ed01049e 2024-09-15T05:07:44.187 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.186+0000 7f0b92729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:44.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.188+0000 7f0b92729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:44.190 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.189+0000 7f0b92729440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:44.190 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.189+0000 7f0b92729440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:07:44.190 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.189+0000 7f0b92729440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:07:44.190 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.189+0000 7f0b92729440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:07:44.262 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:44.262+0000 7f0b92729440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-15T05:07:50.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-15T05:07:50.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:07:50.408 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T05:07:50.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T05:07:50.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:07:50.895 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T05:07:50.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T05:07:50.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:07:50.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:07:50.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:07:50.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:07:50.917 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:50.916+0000 7f3906a0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:50.919 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:50.918+0000 7f3906a0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:50.920 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:50.919+0000 7f3906a0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:07:51.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T05:07:51.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:07:51.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T05:07:51.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:07:51.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:07:51.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:51.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:07:51.247 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:07:51.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:51.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:51.454 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:51.454+0000 7f3906a0c440 -1 Falling back to public interface 2024-09-15T05:07:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:52.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:52.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:52.623 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:07:52.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:07:52.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:52.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:52.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:53.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:53.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:53.983 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:07:53.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:07:53.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:53.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:54.004 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:07:54.004+0000 7f3906a0c440 -1 osd.4 0 log_to_monitors true 2024-09-15T05:07:54.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:55.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:55.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:55.457 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:07:55.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:07:55.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:55.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:55.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:56.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:56.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:56.947 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:07:56.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:07:56.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:56.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:57.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:58.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:58.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:58.333 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:07:58.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:07:58.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:58.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:07:58.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:07:59.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:07:59.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:07:59.742 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:07:59.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:07:59.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:07:59.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:00.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:01.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:01.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:01.105 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:08:01.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:08:01.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:01.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:01.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:02.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:02.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:02.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:08:02.494 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:08:02.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:02.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:02.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:03.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:03.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:03.951 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:08:03.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:08:03.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:03.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:04.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:05.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:05.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:05.348 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:08:05.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:08:05.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:05.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:05.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:06.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:06.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:06.805 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:08:06.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:08:06.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:06.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:07.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:08.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:08.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:08.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:08:08.600 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:08:08.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:08.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:09.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:10.394 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:08:10.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:08:10.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:10.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:11.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:12.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:12.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:12.600 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:08:12.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:08:12.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:12.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:13.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:14.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:14.360 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:08:14.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:08:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:14.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:14.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:15.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:15.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:15.950 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:08:15.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:08:15.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:15.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:16.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:17.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:17.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:17.672 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:08:17.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:08:17.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:17.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:08:18.118 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/328482,v1:127.0.0.1:6835/328482] [v2:127.0.0.1:6836/328482,v1:127.0.0.1:6837/328482] exists,up 492f11f2-9231-4583-b663-cec5ed01049e 2024-09-15T05:08:18.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:08:18.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:08:18.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:08:18.124 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:08:18.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:08:18.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=df6eefb0-79dd-4c03-a5a6-29543bffb70e 2024-09-15T05:08:18.127 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 df6eefb0-79dd-4c03-a5a6-29543bffb70e 2024-09-15T05:08:18.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 df6eefb0-79dd-4c03-a5a6-29543bffb70e' 2024-09-15T05:08:18.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:08:18.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCa+Zmdj0TCBAAycCt6JlAyUsdGbdFKZT4FQ== 2024-09-15T05:08:18.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCa+Zmdj0TCBAAycCt6JlAyUsdGbdFKZT4FQ=="}' 2024-09-15T05:08:18.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new df6eefb0-79dd-4c03-a5a6-29543bffb70e -i td/osd-recovery-stats/5/new.json 2024-09-15T05:08:18.603 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:08:18.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-15T05:08:18.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCa+Zmdj0TCBAAycCt6JlAyUsdGbdFKZT4FQ== --osd-uuid df6eefb0-79dd-4c03-a5a6-29543bffb70e 2024-09-15T05:08:18.638 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.638+0000 7fd604408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:18.640 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.639+0000 7fd604408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:18.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.640+0000 7fd604408440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:18.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.641+0000 7fd604408440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:08:18.641 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.641+0000 7fd604408440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:08:18.642 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.641+0000 7fd604408440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:08:18.726 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:18.726+0000 7fd604408440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-15T05:08:24.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-15T05:08:24.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:08:24.840 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:08:24.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:08:24.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:08:25.339 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:08:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:08:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:08:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:08:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:08:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:08:25.362 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:25.361+0000 7ff368762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:25.364 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:25.363+0000 7ff368762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:25.365 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:25.364+0000 7ff368762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:08:25.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:25.689 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:08:25.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:08:25.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:25.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:25.880 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:25.879+0000 7ff368762440 -1 Falling back to public interface 2024-09-15T05:08:26.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:27.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:27.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:27.050 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:08:27.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:08:27.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:27.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:27.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:28.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:28.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:28.403 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:08:28.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:08:28.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:28.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:28.507 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:08:28.506+0000 7ff368762440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:08:28.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:29.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:29.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:29.867 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:08:29.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:08:29.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:29.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:30.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:31.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:31.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:31.367 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:08:31.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:08:31.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:31.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:31.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:32.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:32.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:32.754 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:08:32.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:08:32.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:32.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:33.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:34.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:34.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:34.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:08:34.152 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:08:34.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:34.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:34.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:35.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:35.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:35.666 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:08:35.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:08:35.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:35.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:36.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:37.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:37.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:37.073 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:08:37.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:08:37.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:37.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:37.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:38.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:38.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:38.520 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:08:38.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:08:38.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:38.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:39.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:40.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:40.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:40.019 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:08:40.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:08:40.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:40.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:40.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:41.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:41.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:41.522 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:08:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:08:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:42.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:43.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:43.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:43.079 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:08:43.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:08:43.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:43.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:43.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:44.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:44.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:44.701 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:08:44.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:08:44.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:44.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:45.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:46.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:46.662 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:08:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:08:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:46.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:47.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:48.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:48.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:48.430 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:08:48.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:08:48.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:48.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:49.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:50.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:50.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:50.267 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:08:50.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:08:50.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:50.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:50.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:51.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:51.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:51.953 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:08:51.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:08:51.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:51.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:52.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:08:53.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:08:53.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:08:53.475 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:08:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:08:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:08:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:08:53.845 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/330236,v1:127.0.0.1:6843/330236] [v2:127.0.0.1:6844/330236,v1:127.0.0.1:6845/330236] exists,up df6eefb0-79dd-4c03-a5a6-29543bffb70e 2024-09-15T05:08:53.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:08:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:08:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:08:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-09-15T05:08:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T05:08:54.383 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:08:54.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:08:55.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-09-15T05:08:56.186 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T05:08:56.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-09-15T05:08:57.102 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 1 2024-09-15T05:08:57.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-09-15T05:08:57.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:08:57.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:08:57.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:08:57.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:08:57.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:08:57.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:08:57.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:08:57.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:08:57.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:08:57.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:08:57.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:57.604 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:08:57.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-15T05:08:57.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-15T05:08:57.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T05:08:57.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:57.752 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:08:57.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-15T05:08:57.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-15T05:08:57.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-15T05:08:57.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:57.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:08:57.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509469 2024-09-15T05:08:57.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509469 2024-09-15T05:08:57.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509469' 2024-09-15T05:08:57.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:57.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:08:58.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345941 2024-09-15T05:08:58.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345941 2024-09-15T05:08:58.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509469 3-85899345941' 2024-09-15T05:08:58.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:58.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:08:58.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182415 2024-09-15T05:08:58.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182415 2024-09-15T05:08:58.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509469 3-85899345941 4-107374182415' 2024-09-15T05:08:58.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:08:58.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:08:58.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509469 3-85899345941 4-107374182415 5-128849018887' 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:08:58.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:08:58.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-15T05:08:58.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:08:58.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-15T05:08:58.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T05:08:58.279 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836524 2024-09-15T05:08:58.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:08:58.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836522 -lt 21474836524 2024-09-15T05:08:58.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:08:59.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:08:59.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:09:00.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836525 -lt 21474836524 2024-09-15T05:09:00.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:00.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-15T05:09:00.038 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:00.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:09:00.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-15T05:09:00.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:00.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-15T05:09:00.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-15T05:09:00.040 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672996 2024-09-15T05:09:00.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:09:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672997 -lt 42949672996 2024-09-15T05:09:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:00.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509469 2024-09-15T05:09:00.405 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:00.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:09:00.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509469 2024-09-15T05:09:00.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:00.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509469 2024-09-15T05:09:00.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-15T05:09:00.408 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509469 2024-09-15T05:09:00.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:09:00.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509470 -lt 64424509469 2024-09-15T05:09:00.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:00.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345941 2024-09-15T05:09:00.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:00.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:09:00.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345941 2024-09-15T05:09:00.757 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:00.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345941 2024-09-15T05:09:00.758 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345941 2024-09-15T05:09:00.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-09-15T05:09:00.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:09:01.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345941 2024-09-15T05:09:01.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:01.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182415 2024-09-15T05:09:01.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:01.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:09:01.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182415 2024-09-15T05:09:01.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:01.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182415 2024-09-15T05:09:01.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-15T05:09:01.111 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182415 2024-09-15T05:09:01.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:09:01.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182416 -lt 107374182415 2024-09-15T05:09:01.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:01.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018887 2024-09-15T05:09:01.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:01.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:09:01.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018887 2024-09-15T05:09:01.463 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:01.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-15T05:09:01.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-15T05:09:01.465 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018887 2024-09-15T05:09:01.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:09:01.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018887 -lt 128849018887 2024-09-15T05:09:01.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:09:01.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:09:01.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:09:02.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:09:02.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:09:02.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:09:02.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:09:02.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:09:02.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:09:02.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:09:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:09:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:09:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:09:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-09-15T05:09:03.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-09-15T05:09:03.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:09:03.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:09:03.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:09:03.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:09:03.599 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:09:03.600 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:09:03.600 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:09:03.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T05:09:03.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-15T05:09:03.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T05:09:04.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-09-15T05:09:04.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-09-15T05:09:04.968 INFO:tasks.workunit.client.0.smithi138.stderr:noout is set 2024-09-15T05:09:04.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-09-15T05:09:05.730 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:09:05.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-09-15T05:09:05.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 321465 2024-09-15T05:09:05.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-09-15T05:09:06.192 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:09:06.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-09-15T05:09:06.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-09-15T05:09:06.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-09-15T05:09:06.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:06.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-09-15T05:09:06.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:06.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-09-15T05:09:07.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-09-15T05:09:07.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-09-15T05:09:07.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-09-15T05:09:07.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-09-15T05:09:07.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-09-15T05:09:07.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-09-15T05:09:07.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-09-15T05:09:07.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-09-15T05:09:07.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:07.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-09-15T05:09:08.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-09-15T05:09:08.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-09-15T05:09:08.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-09-15T05:09:08.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-09-15T05:09:08.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-09-15T05:09:08.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-09-15T05:09:08.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-09-15T05:09:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-09-15T05:09:08.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-09-15T05:09:08.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:08.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-09-15T05:09:09.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-09-15T05:09:09.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-09-15T05:09:09.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-09-15T05:09:09.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-09-15T05:09:09.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-09-15T05:09:09.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-09-15T05:09:09.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-09-15T05:09:09.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-09-15T05:09:09.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:09.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-09-15T05:09:10.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-09-15T05:09:10.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-09-15T05:09:10.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-09-15T05:09:10.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-09-15T05:09:10.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-09-15T05:09:10.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-09-15T05:09:10.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-09-15T05:09:10.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-09-15T05:09:10.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-09-15T05:09:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:10.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-09-15T05:09:11.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-09-15T05:09:11.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-09-15T05:09:11.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-09-15T05:09:11.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-09-15T05:09:11.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-09-15T05:09:11.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-09-15T05:09:11.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-09-15T05:09:11.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-09-15T05:09:11.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:11.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-09-15T05:09:12.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-09-15T05:09:12.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-09-15T05:09:12.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-09-15T05:09:12.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-09-15T05:09:12.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-09-15T05:09:12.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-09-15T05:09:12.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-09-15T05:09:12.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-09-15T05:09:12.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-09-15T05:09:12.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:12.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-09-15T05:09:13.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-09-15T05:09:13.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-09-15T05:09:13.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-09-15T05:09:13.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-09-15T05:09:13.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-09-15T05:09:13.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-09-15T05:09:13.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-09-15T05:09:13.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-09-15T05:09:13.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:13.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-09-15T05:09:14.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-09-15T05:09:14.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-09-15T05:09:14.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-09-15T05:09:14.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-09-15T05:09:14.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-09-15T05:09:14.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-09-15T05:09:14.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-09-15T05:09:14.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-09-15T05:09:14.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-09-15T05:09:14.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:14.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-09-15T05:09:15.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-09-15T05:09:15.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-09-15T05:09:15.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-09-15T05:09:15.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-09-15T05:09:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-09-15T05:09:15.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-09-15T05:09:15.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-09-15T05:09:15.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-09-15T05:09:15.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:15.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-09-15T05:09:16.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-09-15T05:09:16.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-09-15T05:09:16.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-09-15T05:09:16.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-09-15T05:09:16.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-09-15T05:09:16.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-09-15T05:09:16.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-09-15T05:09:16.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-09-15T05:09:16.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:16.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-09-15T05:09:17.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:17.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-09-15T05:09:17.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:17.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-09-15T05:09:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-09-15T05:09:17.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-09-15T05:09:17.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-09-15T05:09:17.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 323220 2024-09-15T05:09:17.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-09-15T05:09:18.015 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:09:18.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:09:18.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:09:18.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:09:18.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:09:18.033 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:09:18.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:09:18.051 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:18.051+0000 7fe9e3929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:18.053 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:18.053+0000 7fe9e3929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:18.054 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:18.054+0000 7fe9e3929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:18.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:09:18.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:09:18.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:09:18.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:18.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:09:18.576 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:18.575+0000 7fe9e3929440 -1 Falling back to public interface 2024-09-15T05:09:18.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:19.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:19.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:19.817 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:09:19.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:09:19.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:19.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:09:20.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:21.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:21.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:21.237 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:09:21.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:09:21.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:21.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:09:21.403 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:21.402+0000 7fe9e3929440 -1 osd.0 39 log_to_monitors true 2024-09-15T05:09:21.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:22.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:22.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:22.609 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:09:22.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:09:22.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:22.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:09:22.957 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 44 last_clean_interval [5,39) [v2:127.0.0.1:6802/333920,v1:127.0.0.1:6803/333920] [v2:127.0.0.1:6804/333920,v1:127.0.0.1:6805/333920] exists,up 1b11038f-c338-42a9-9887-52710ad0e257 2024-09-15T05:09:22.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:09:22.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:09:22.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:09:22.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-09-15T05:09:25.960 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-09-15T05:09:25.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-09-15T05:09:25.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:25.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-09-15T05:09:26.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-09-15T05:09:26.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-09-15T05:09:26.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-09-15T05:09:26.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-09-15T05:09:26.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-09-15T05:09:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-09-15T05:09:26.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-09-15T05:09:26.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-09-15T05:09:26.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:26.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-09-15T05:09:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-09-15T05:09:27.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-09-15T05:09:27.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-09-15T05:09:27.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-09-15T05:09:27.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-09-15T05:09:27.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-09-15T05:09:27.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-09-15T05:09:27.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-09-15T05:09:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-09-15T05:09:28.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-09-15T05:09:28.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-09-15T05:09:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-09-15T05:09:28.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-09-15T05:09:28.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-09-15T05:09:28.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-09-15T05:09:28.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-09-15T05:09:28.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-09-15T05:09:28.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-09-15T05:09:28.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:28.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-09-15T05:09:29.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-09-15T05:09:29.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-09-15T05:09:29.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-09-15T05:09:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-09-15T05:09:29.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-09-15T05:09:29.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-09-15T05:09:29.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-09-15T05:09:29.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-09-15T05:09:29.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:29.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-09-15T05:09:30.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-09-15T05:09:30.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-09-15T05:09:30.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-09-15T05:09:30.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-09-15T05:09:30.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-09-15T05:09:30.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-09-15T05:09:30.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-09-15T05:09:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-09-15T05:09:30.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:30.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-09-15T05:09:31.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-09-15T05:09:31.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-09-15T05:09:31.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-09-15T05:09:31.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-09-15T05:09:31.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-09-15T05:09:31.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-09-15T05:09:31.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-09-15T05:09:31.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-09-15T05:09:31.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-09-15T05:09:31.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:31.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-09-15T05:09:32.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-09-15T05:09:32.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-09-15T05:09:32.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-09-15T05:09:32.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-09-15T05:09:32.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-09-15T05:09:32.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-09-15T05:09:32.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-09-15T05:09:32.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-09-15T05:09:32.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:32.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-09-15T05:09:33.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-09-15T05:09:33.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-09-15T05:09:33.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-09-15T05:09:33.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-09-15T05:09:33.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-09-15T05:09:33.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-09-15T05:09:33.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-09-15T05:09:33.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-09-15T05:09:33.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:33.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-09-15T05:09:34.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-09-15T05:09:34.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-09-15T05:09:34.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-09-15T05:09:34.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-09-15T05:09:34.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-09-15T05:09:34.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-09-15T05:09:34.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-09-15T05:09:34.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-09-15T05:09:34.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-09-15T05:09:34.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:34.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-09-15T05:09:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-09-15T05:09:35.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-09-15T05:09:35.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-09-15T05:09:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-09-15T05:09:35.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-09-15T05:09:35.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-09-15T05:09:35.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-09-15T05:09:35.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-09-15T05:09:35.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:35.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-09-15T05:09:36.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-09-15T05:09:36.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-09-15T05:09:36.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-09-15T05:09:36.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-09-15T05:09:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-09-15T05:09:36.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-09-15T05:09:36.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-09-15T05:09:36.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-09-15T05:09:36.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:09:36.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:09:36.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:09:36.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:09:37.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-09-15T05:09:37.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-09-15T05:09:37.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:09:37.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-09-15T05:09:37.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-09-15T05:09:37.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:09:37.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-09-15T05:09:37.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-09-15T05:09:37.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:09:37.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-09-15T05:09:37.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-09-15T05:09:37.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-09-15T05:09:37.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-09-15T05:09:37.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-09-15T05:09:38.514 INFO:tasks.workunit.client.0.smithi138.stderr:noout is unset 2024-09-15T05:09:38.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-09-15T05:09:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already out. osd.2 is already out. 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:09:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:09:39.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:09:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:09:39.093 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:09:39.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:09:39.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:09:39.112 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:39.111+0000 7f4678008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:39.117 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:39.115+0000 7f4678008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:39.118 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:39.116+0000 7f4678008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:39.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:09:39.532 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:09:39.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:39.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:09:39.643 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:39.642+0000 7f4678008440 -1 Falling back to public interface 2024-09-15T05:09:40.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:41.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:41.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:41.010 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:09:41.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:09:41.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:41.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:09:41.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:42.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:42.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:42.371 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:09:42.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:09:42.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:42.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:09:42.635 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:09:42.634+0000 7f4678008440 -1 osd.1 45 log_to_monitors true 2024-09-15T05:09:42.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:09:43.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:09:43.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:09:43.857 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:09:43.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:09:43.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:09:43.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:09:44.212 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up out weight 0 up_from 58 up_thru 44 down_at 46 last_clean_interval [10,45) [v2:127.0.0.1:6810/336622,v1:127.0.0.1:6811/336622] [v2:127.0.0.1:6812/336622,v1:127.0.0.1:6813/336622] exists,up dbe270c0-a63a-459c-9ca9-f16fca156e74 2024-09-15T05:09:44.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:09:44.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:09:44.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:09:44.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-09-15T05:09:47.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-09-15T05:09:48.030 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 4 2024-09-15T05:09:48.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-09-15T05:09:49.799 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:09:49.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-09-15T05:09:49.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:09:49.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:09:49.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:09:49.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:09:50.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-09-15T05:09:50.410 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:09:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-09-15T05:09:52.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-09-15T05:09:52.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:09:52.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:09:52.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:09:52.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:09:52.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:09:52.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:09:52.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:09:52.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:09:52.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:09:52.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:09:52.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:09:52.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:09:52.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:09:52.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:09:52.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:09:52.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:09:52.872 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:52.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:09:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=210453397513 2024-09-15T05:09:53.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 210453397513 2024-09-15T05:09:53.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513' 2024-09-15T05:09:53.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:53.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:09:53.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=249108103172 2024-09-15T05:09:53.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 249108103172 2024-09-15T05:09:53.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513 1-249108103172' 2024-09-15T05:09:53.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:53.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:09:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509482 2024-09-15T05:09:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509482 2024-09-15T05:09:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509482' 2024-09-15T05:09:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:53.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:09:53.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345954 2024-09-15T05:09:53.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345954 2024-09-15T05:09:53.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509482 3-85899345954' 2024-09-15T05:09:53.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:53.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:09:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182429 2024-09-15T05:09:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182429 2024-09-15T05:09:53.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509482 3-85899345954 4-107374182429' 2024-09-15T05:09:53.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:09:53.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:09:53.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018900 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018900 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509482 3-85899345954 4-107374182429 5-128849018900' 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-210453397513 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:09:53.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-210453397513 2024-09-15T05:09:53.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:53.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=210453397513 2024-09-15T05:09:53.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 210453397513' 2024-09-15T05:09:53.585 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 210453397513 2024-09-15T05:09:53.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:09:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 210453397513 -lt 210453397513 2024-09-15T05:09:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:53.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-249108103172 2024-09-15T05:09:53.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:53.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:09:53.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-249108103172 2024-09-15T05:09:53.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:53.965 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 249108103172 2024-09-15T05:09:53.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=249108103172 2024-09-15T05:09:53.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 249108103172' 2024-09-15T05:09:53.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:09:54.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 249108103172 -lt 249108103172 2024-09-15T05:09:54.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:54.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509482 2024-09-15T05:09:54.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:54.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:09:54.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509482 2024-09-15T05:09:54.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:54.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509482 2024-09-15T05:09:54.443 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509482 2024-09-15T05:09:54.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-09-15T05:09:54.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:09:54.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509483 -lt 64424509482 2024-09-15T05:09:54.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:54.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345954 2024-09-15T05:09:54.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:54.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:09:54.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345954 2024-09-15T05:09:54.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:54.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345954 2024-09-15T05:09:54.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2024-09-15T05:09:54.872 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345954 2024-09-15T05:09:54.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:09:55.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345955 -lt 85899345954 2024-09-15T05:09:55.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:55.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182429 2024-09-15T05:09:55.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:55.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:09:55.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182429 2024-09-15T05:09:55.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:55.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182429 2024-09-15T05:09:55.358 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182429 2024-09-15T05:09:55.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182429' 2024-09-15T05:09:55.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:09:55.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182429 -lt 107374182429 2024-09-15T05:09:55.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:09:55.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018900 2024-09-15T05:09:55.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:09:55.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018900 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018900 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018900 2024-09-15T05:09:55.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:09:56.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018900 -lt 128849018900 2024-09-15T05:09:56.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:09:56.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:09:56.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:09:57.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:09:57.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:09:57.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:09:57.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:09:57.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:09:57.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:09:57.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:09:57.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:09:57.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:09:57.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:09:57.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:09:57.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:09:58.029 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:09:58.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:09:58.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:09:58.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:09:58.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:09:58.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:09:58.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:09:58.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:09:58.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:09:59.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:09:59.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:09:59.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:09:59.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:09:59.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:09:59.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:00.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:00.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:00.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:00.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:00.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:00.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:01.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-15T05:10:01.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-15T05:10:01.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:01.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:01.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:01.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:01.217 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:01.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:01.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:01.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:01.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:02.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:02.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:02.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:02.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:02.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:02.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:02.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:02.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:02.216 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:02.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:10:02.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:10:02.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:02.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:02.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:02.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:03.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:03.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:03.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:03.353 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:04.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:04.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:04.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:04.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:04.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:04.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:04.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:04.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:04.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:04.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:10:04.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:10:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:04.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:04.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:04.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:05.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:05.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:05.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:05.082 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:05.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:06.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:10:06.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:10:06.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:06.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:06.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:06.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:06.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:06.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:06.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:06.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:06.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:07.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:07.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:07.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:10:07.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:10:07.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:07.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:07.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:07.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:07.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:07.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:07.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:07.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:07.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:07.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:07.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:08.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:08.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:08.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:08.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:08.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:08.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:08.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:08.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:09.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:10:09.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:10:09.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:09.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:09.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:09.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:09.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:09.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:09.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:09.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:09.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:09.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:09.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:09.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:09.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:09.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:09.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:10.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:10.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:10.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:10.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:10.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:10.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:10.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:10.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:10.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:10.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:10:10.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:10:10.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:10.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:10.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:10.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:10.882 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:11.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:11.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:11.237 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:11.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:11.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:11.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:11.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:11.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:11.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:11.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:11.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:11.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:10:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:10:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:12.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:12.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:12.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:12.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:13.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:13.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:13.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:13.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:13.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:13.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:13.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:13.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:13.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:13.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:13.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:13.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:14.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:10:14.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:10:14.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:14.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:14.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:14.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:14.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:14.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:14.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:14.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:14.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:15.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:15.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:15.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:15.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:15.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:10:15.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:10:15.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:15.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:15.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:16.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:16.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:16.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:16.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:16.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:16.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:16.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:16.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:17.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:10:17.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:10:17.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:17.350 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:17.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:17.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:17.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:17.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:18.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:19.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-09-15T05:10:19.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-09-15T05:10:19.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:19.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:19.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:19.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:19.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:20.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:20.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:20.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:20.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:20.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:10:20.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:10:20.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:20.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:20.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:20.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:20.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:20.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:20.677 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:20.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:20.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:20.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:20.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:21.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:10:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:21.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:10:21.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:10:22.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:10:22.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:10:22.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:10:22.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:10:22.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:10:22.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:10:22.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:10:22.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:10:22.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:10:22.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:10:22.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:10:22.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:10:22.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:10:22.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:10:22.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:10:22.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-09-15T05:10:22.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:10:22.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:10:22.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:10:22.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:10:23.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-09-15T05:10:23.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-09-15T05:10:23.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:10:23.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:10:23.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-09-15T05:10:23.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:10:23.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:10:23.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-09-15T05:10:23.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:10:23.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.298 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.298 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-15T05:10:23.299 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:10:23.299 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:10:23.299 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-09-15T05:10:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:10:23.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:10:23.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:10:23.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:10:23.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:10:23.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:10:23.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:10:23.318 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-09-15T05:10:23.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-09-15T05:10:23.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-09-15T05:10:23.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:10:23.323 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:10:23.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-15T05:10:23.336 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-09-15T05:10:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-09-15T05:10:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-15T05:10:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:10:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-15T05:10:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-15T05:10:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:10:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:10:23.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:10:23.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:10:23.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-09-15T05:10:23.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:10:23.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:10:23.858 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:10:23.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-09-15T05:10:23.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:10:23.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:10:23.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:10:23.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:10:23.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:10:25.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:10:25.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:10:25.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:10:25.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:10:25.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:10:25.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:10:25.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:10:25.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:10:25.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:10:25.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:10:25.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:10:25.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:10:25.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:10:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:10:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:10:25.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:10:25.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:10:25.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:10:25.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:10:25.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:10:25.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:10:25.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:10:25.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:10:25.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:10:25.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:10:25.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:10:25.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:10:25.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:10:25.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:10:25.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:10:25.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:10:25.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:10:25.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:10:25.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:10:25.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:10:25.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:10:25.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:10:25.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:10:25.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:10:25.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:10:25.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:10:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:10:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:10:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:10:25.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:10:25.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:10:25.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:10:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:10:25.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:10:25.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:10:25.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:10:25.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:10:25.648 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:10:25.648 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.648 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:25.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:10:25.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:10:25.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:10:25.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:10:25.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:10:25.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:10:25.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:10:25.909 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:10:25.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:10:25.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:10:25.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:10:25.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:10:26.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:10:26.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:10:26.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:26.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:10:26.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:10:26.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:10:26.940 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:10:26.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:10:26.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:10:26.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:10:26.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e1e129da-78ab-4a73-b5ee-84981bf80066 2024-09-15T05:10:26.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e1e129da-78ab-4a73-b5ee-84981bf80066' 2024-09-15T05:10:26.943 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 e1e129da-78ab-4a73-b5ee-84981bf80066 2024-09-15T05:10:26.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:10:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCbOZmrc/YOBAA/eMAczGb5H9S+ZPPYX0Yfg== 2024-09-15T05:10:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCbOZmrc/YOBAA/eMAczGb5H9S+ZPPYX0Yfg=="}' 2024-09-15T05:10:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e1e129da-78ab-4a73-b5ee-84981bf80066 -i td/osd-recovery-stats/0/new.json 2024-09-15T05:10:27.253 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:10:27.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:10:27.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCbOZmrc/YOBAA/eMAczGb5H9S+ZPPYX0Yfg== --osd-uuid e1e129da-78ab-4a73-b5ee-84981bf80066 2024-09-15T05:10:27.287 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.287+0000 7efeb9a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:27.299 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.298+0000 7efeb9a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.300+0000 7efeb9a15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.300+0000 7efeb9a15440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:10:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.300+0000 7efeb9a15440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:10:27.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.300+0000 7efeb9a15440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:10:27.339 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:27.338+0000 7efeb9a15440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:10:32.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:10:32.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:10:32.633 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:10:32.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:10:32.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:10:33.113 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:10:33.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:10:33.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:10:33.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:10:33.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:10:33.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:10:33.135 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:33.134+0000 7f52fbc35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:33.154 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:33.153+0000 7f52fbc35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:33.155 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:33.154+0000 7f52fbc35440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:10:33.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:10:33.568 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:10:33.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:33.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:33.683 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:33.683+0000 7f52fbc35440 -1 Falling back to public interface 2024-09-15T05:10:33.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:34.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:34.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:34.916 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:10:34.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:10:34.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:34.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:35.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:36.132 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:10:36.131+0000 7f52fbc35440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:10:36.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:36.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:36.275 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:10:36.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:10:36.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:36.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:36.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:37.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:37.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:37.783 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:10:37.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:10:37.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:37.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:38.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:39.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:39.142 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:10:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:10:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:39.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:39.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:40.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:40.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:40.512 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:10:40.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:10:40.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:40.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:40.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:42.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:42.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:42.001 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:10:42.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:10:42.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:42.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:42.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:43.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:43.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:43.362 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:10:43.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:10:43.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:43.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:43.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:44.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:44.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:44.768 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:10:44.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:10:44.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:44.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:45.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:10:46.154 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:10:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:46.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:47.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:47.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:47.542 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:10:47.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:10:47.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:47.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:48.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:49.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:49.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:49.139 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:10:49.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:10:49.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:49.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:49.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:50.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:50.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:50.763 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:10:50.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:10:50.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:50.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:51.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:52.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:52.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:52.573 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:10:52.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:10:52.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:52.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:53.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:54.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:54.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:54.277 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:10:54.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:10:54.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:54.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:54.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:55.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:55.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:55.991 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:10:55.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:10:55.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:55.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:56.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:57.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:57.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:57.829 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:10:57.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:10:57.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:57.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:58.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:10:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:10:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:10:59.571 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:10:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:10:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:10:59.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:10:59.964 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/340198,v1:127.0.0.1:6803/340198] [v2:127.0.0.1:6804/340198,v1:127.0.0.1:6805/340198] exists,up e1e129da-78ab-4a73-b5ee-84981bf80066 2024-09-15T05:10:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:10:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:10:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:10:59.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:10:59.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:10:59.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28d356a5-23af-43f2-9537-3102296fc8e1 2024-09-15T05:10:59.972 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 28d356a5-23af-43f2-9537-3102296fc8e1 2024-09-15T05:10:59.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 28d356a5-23af-43f2-9537-3102296fc8e1' 2024-09-15T05:10:59.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:10:59.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjbOZmB0WJOhAAaLfpDY5JVlM/fxaf3o6hjA== 2024-09-15T05:10:59.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjbOZmB0WJOhAAaLfpDY5JVlM/fxaf3o6hjA=="}' 2024-09-15T05:10:59.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28d356a5-23af-43f2-9537-3102296fc8e1 -i td/osd-recovery-stats/1/new.json 2024-09-15T05:11:00.563 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:11:00.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:11:00.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjbOZmB0WJOhAAaLfpDY5JVlM/fxaf3o6hjA== --osd-uuid 28d356a5-23af-43f2-9537-3102296fc8e1 2024-09-15T05:11:00.599 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.598+0000 7f040a409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:00.623 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.622+0000 7f040a409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:00.624 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.623+0000 7f040a409440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:00.624 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.623+0000 7f040a409440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:11:00.625 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.623+0000 7f040a409440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:11:00.625 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.623+0000 7f040a409440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:11:00.707 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:00.706+0000 7f040a409440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:11:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:11:06.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:11:06.413 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:11:06.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:11:06.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:11:06.899 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:11:06.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:11:06.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:11:06.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:11:06.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:11:06.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:11:06.921 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:06.920+0000 7f576c62d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:06.923 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:06.923+0000 7f576c62d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:06.924 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:06.924+0000 7f576c62d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:11:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:11:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:11:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:11:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:11:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:11:07.249 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:11:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:07.441 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:07.440+0000 7f576c62d440 -1 Falling back to public interface 2024-09-15T05:11:07.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:08.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:08.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:08.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:11:08.619 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:11:08.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:08.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:08.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:09.908 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:09.906+0000 7f576c62d440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:11:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:09.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:11:09.972 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:11:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:10.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:11.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:11.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:11.590 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:11:11.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:11:11.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:11.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:11.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:12.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:12.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:12.990 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:11:12.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:11:12.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:12.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:13.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:14.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:14.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:14.439 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:11:14.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:11:14.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:14.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:14.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:15.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:15.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:15.845 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:11:15.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:11:15.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:15.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:16.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:17.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:17.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:17.218 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:11:17.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:11:17.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:17.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:17.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:18.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:18.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:18.684 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:11:18.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:11:18.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:18.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:19.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:20.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:20.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:20.072 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:11:20.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:11:20.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:20.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:20.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:21.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:21.452 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:11:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:11:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:21.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:21.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:22.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:22.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:22.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:11:22.981 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:11:22.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:22.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:23.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:24.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:24.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:24.950 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:11:24.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:11:24.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:24.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:25.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:26.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:26.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:26.744 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:11:26.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:11:26.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:26.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:27.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:28.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:28.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:28.859 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:11:28.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:11:28.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:28.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:30.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:30.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:30.776 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:11:30.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:11:30.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:30.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:31.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:32.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:32.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:32.696 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:11:32.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:11:32.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:32.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:33.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:34.188 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:11:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:11:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:11:34.533 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/341953,v1:127.0.0.1:6811/341953] [v2:127.0.0.1:6812/341953,v1:127.0.0.1:6813/341953] exists,up 28d356a5-23af-43f2-9537-3102296fc8e1 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:11:34.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:11:34.539 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:11:34.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=54edb22c-e7d4-4278-8b25-e90bc029e615 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 54edb22c-e7d4-4278-8b25-e90bc029e615 2024-09-15T05:11:34.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 54edb22c-e7d4-4278-8b25-e90bc029e615' 2024-09-15T05:11:34.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:11:34.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGbOZmzsHVIBAAqR0Mb1mMeS6yrn5Pcx/fjw== 2024-09-15T05:11:34.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGbOZmzsHVIBAAqR0Mb1mMeS6yrn5Pcx/fjw=="}' 2024-09-15T05:11:34.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 54edb22c-e7d4-4278-8b25-e90bc029e615 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:11:34.981 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:11:34.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:11:34.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGbOZmzsHVIBAAqR0Mb1mMeS6yrn5Pcx/fjw== --osd-uuid 54edb22c-e7d4-4278-8b25-e90bc029e615 2024-09-15T05:11:35.016 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.015+0000 7fead400d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:35.018 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.017+0000 7fead400d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:35.019 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.018+0000 7fead400d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:35.020 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.019+0000 7fead400d440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:11:35.020 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.019+0000 7fead400d440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:11:35.020 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.019+0000 7fead400d440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:11:35.079 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:35.079+0000 7fead400d440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:11:40.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:11:40.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:11:40.668 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:11:40.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:11:40.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:11:41.148 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:11:41.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:11:41.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:11:41.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:11:41.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:11:41.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:11:41.170 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:41.169+0000 7f2908834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:41.176 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:41.175+0000 7f2908834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:41.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:41.176+0000 7f2908834440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:11:41.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:11:41.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:11:41.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:11:41.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:41.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:41.710 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:41.710+0000 7f2908834440 -1 Falling back to public interface 2024-09-15T05:11:41.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:42.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:42.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:42.950 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:11:42.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:11:42.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:42.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:43.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:44.215 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:11:44.214+0000 7f2908834440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:11:44.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:44.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:44.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:11:44.308 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:11:44.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:44.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:44.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:45.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:45.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:45.884 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:11:45.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:11:45.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:45.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:46.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:47.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:47.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:47.271 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:11:47.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:11:47.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:47.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:47.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:48.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:48.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:48.704 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:11:48.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:11:48.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:48.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:49.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:50.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:50.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:50.083 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:11:50.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:11:50.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:50.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:50.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:51.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:51.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:51.445 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:11:51.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:11:51.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:51.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:51.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:52.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:52.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:52.955 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:11:52.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:11:52.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:52.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:53.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:54.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:54.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:54.526 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:11:54.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:11:54.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:54.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:54.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:55.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:55.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:55.983 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:11:55.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:11:55.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:55.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:56.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:57.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:57.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:57.513 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:11:57.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:11:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:57.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:11:58.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:11:59.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:11:59.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:11:59.288 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:11:59.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:11:59.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:11:59.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:00.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:01.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:01.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:01.338 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:12:01.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:12:01.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:01.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:02.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:03.131 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:12:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:12:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:04.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:04.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:04.950 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:12:04.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:12:04.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:04.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:05.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:06.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:06.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:06.799 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:12:06.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:12:06.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:06.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:07.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:08.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:08.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:08.174 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:12:08.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:12:08.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:08.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:12:08.552 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/343707,v1:127.0.0.1:6819/343707] [v2:127.0.0.1:6820/343707,v1:127.0.0.1:6821/343707] exists,up 54edb22c-e7d4-4278-8b25-e90bc029e615 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:12:08.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:12:08.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:12:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d82605eb-5aee-4279-b15d-b321a6285cf0 2024-09-15T05:12:08.560 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 d82605eb-5aee-4279-b15d-b321a6285cf0 2024-09-15T05:12:08.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d82605eb-5aee-4279-b15d-b321a6285cf0' 2024-09-15T05:12:08.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:12:08.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCobOZmtW75IRAAScQSo+aM7BjkG3UnTbIajg== 2024-09-15T05:12:08.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCobOZmtW75IRAAScQSo+aM7BjkG3UnTbIajg=="}' 2024-09-15T05:12:08.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d82605eb-5aee-4279-b15d-b321a6285cf0 -i td/osd-recovery-stats/3/new.json 2024-09-15T05:12:09.034 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:12:09.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T05:12:09.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCobOZmtW75IRAAScQSo+aM7BjkG3UnTbIajg== --osd-uuid d82605eb-5aee-4279-b15d-b321a6285cf0 2024-09-15T05:12:09.070 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.069+0000 7f3256521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:09.072 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.071+0000 7f3256521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:09.073 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.072+0000 7f3256521440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:09.073 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.072+0000 7f3256521440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:12:09.074 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.073+0000 7f3256521440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:12:09.074 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.073+0000 7f3256521440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:12:09.107 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:09.107+0000 7f3256521440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T05:12:14.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T05:12:14.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:12:14.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:12:14.693 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:12:14.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:12:15.216 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:12:15.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:12:15.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:12:15.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:12:15.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:12:15.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:12:15.239 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:15.238+0000 7f925900d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:15.274 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:15.273+0000 7f925900d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:15.275 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:15.274+0000 7f925900d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:15.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:12:15.578 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:12:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:15.788 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:15.788+0000 7f925900d440 -1 Falling back to public interface 2024-09-15T05:12:15.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:16.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:16.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:16.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:12:16.962 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:12:16.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:16.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:17.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:18.277 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:18.276+0000 7f925900d440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:12:18.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:18.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:18.320 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:12:18.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:12:18.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:18.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:18.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:19.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:19.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:19.905 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:12:19.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:12:19.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:19.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:20.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:21.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:21.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:21.275 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:12:21.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:12:21.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:21.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:21.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:22.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:22.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:22.673 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:12:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:12:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:22.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:23.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:24.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:24.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:24.121 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:12:24.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:12:24.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:24.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:24.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:25.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:25.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:25.527 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:12:25.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:12:25.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:25.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:25.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:26.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:26.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:26.887 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:12:26.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:12:26.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:26.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:27.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:28.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:28.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:28.317 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:12:28.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:12:28.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:28.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:28.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:29.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:29.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:29.695 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:12:29.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:12:29.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:29.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:30.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:31.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:31.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:31.110 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:12:31.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:12:31.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:31.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:31.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:32.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:32.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:32.721 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:12:32.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:12:32.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:32.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:33.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:34.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:34.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:34.440 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:12:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:12:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:35.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:36.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:36.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:36.014 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:12:36.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:12:36.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:36.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:36.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:37.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:37.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:37.727 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:12:37.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:12:37.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:37.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:38.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:39.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:39.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:39.381 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:12:39.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:12:39.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:39.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:40.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:41.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:41.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:41.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:12:41.210 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:12:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:41.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:12:41.662 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/345461,v1:127.0.0.1:6827/345461] [v2:127.0.0.1:6828/345461,v1:127.0.0.1:6829/345461] exists,up d82605eb-5aee-4279-b15d-b321a6285cf0 2024-09-15T05:12:41.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-15T05:12:41.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:12:41.669 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:12:41.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-15T05:12:41.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:12:41.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=49650ca4-ed44-4af4-8f6f-91de6683808d 2024-09-15T05:12:41.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 49650ca4-ed44-4af4-8f6f-91de6683808d' 2024-09-15T05:12:41.671 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 49650ca4-ed44-4af4-8f6f-91de6683808d 2024-09-15T05:12:41.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:12:41.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJbOZmm2h+KBAAik3eHNYOCFI4/AVzDivoSg== 2024-09-15T05:12:41.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJbOZmm2h+KBAAik3eHNYOCFI4/AVzDivoSg=="}' 2024-09-15T05:12:41.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 49650ca4-ed44-4af4-8f6f-91de6683808d -i td/osd-recovery-stats/4/new.json 2024-09-15T05:12:42.105 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:12:42.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-15T05:12:42.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJbOZmm2h+KBAAik3eHNYOCFI4/AVzDivoSg== --osd-uuid 49650ca4-ed44-4af4-8f6f-91de6683808d 2024-09-15T05:12:42.141 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.139+0000 7fc7b2121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:42.142 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.141+0000 7fc7b2121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:42.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.141+0000 7fc7b2121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:42.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.142+0000 7fc7b2121440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:12:42.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.142+0000 7fc7b2121440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:12:42.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.142+0000 7fc7b2121440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:12:42.221 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:42.220+0000 7fc7b2121440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-15T05:12:48.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-15T05:12:48.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:12:48.071 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T05:12:48.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T05:12:48.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:12:48.556 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T05:12:48.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T05:12:48.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:12:48.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:12:48.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:12:48.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:12:48.579 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:48.578+0000 7f5495d2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:48.604 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:48.603+0000 7f5495d2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:48.605 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:48.604+0000 7f5495d2c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:12:48.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T05:12:48.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:12:48.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T05:12:48.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:12:48.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:12:48.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:48.953 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:12:48.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:12:48.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:48.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:49.138 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:49.137+0000 7f5495d2c440 -1 Falling back to public interface 2024-09-15T05:12:49.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:50.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:50.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:50.304 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:12:50.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:12:50.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:50.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:50.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:51.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:51.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:51.686 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:12:51.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:12:51.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:51.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:51.711 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:12:51.710+0000 7f5495d2c440 -1 osd.4 0 log_to_monitors true 2024-09-15T05:12:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:53.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:53.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:53.160 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:12:53.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:12:53.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:53.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:53.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:54.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:54.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:54.551 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:12:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:12:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:55.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:55.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:55.954 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:12:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:12:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:56.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:57.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:57.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:57.489 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:12:57.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:12:57.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:57.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:57.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:12:58.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:12:58.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:12:58.954 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:12:58.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:12:58.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:12:58.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:12:59.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:00.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:00.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:00.421 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:13:00.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:13:00.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:00.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:00.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:01.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:01.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:01.783 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:13:01.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:13:01.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:01.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:02.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:03.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:03.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:03.178 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:13:03.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:13:03.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:03.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:03.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:04.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:04.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:04.626 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:13:04.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:13:04.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:04.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:05.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:06.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:06.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:06.183 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:13:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:13:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:06.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:06.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:07.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:07.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:07.899 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:13:07.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:13:07.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:07.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:08.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:09.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:09.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:09.695 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:13:09.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:13:09.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:09.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:10.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:11.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:11.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:11.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:13:11.509 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:13:11.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:11.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:12.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:13.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:13.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:13.230 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:13:13.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:13:13.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:13.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:14.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:15.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:15.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:15.097 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:13:15.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:13:15.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:15.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:13:15.451 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/347215,v1:127.0.0.1:6835/347215] [v2:127.0.0.1:6836/347215,v1:127.0.0.1:6837/347215] exists,up 49650ca4-ed44-4af4-8f6f-91de6683808d 2024-09-15T05:13:15.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:13:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:13:15.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:13:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:13:15.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c5d6bdc9-c457-47c4-b073-b6c0ecf4327b 2024-09-15T05:13:15.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 c5d6bdc9-c457-47c4-b073-b6c0ecf4327b' 2024-09-15T05:13:15.459 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 c5d6bdc9-c457-47c4-b073-b6c0ecf4327b 2024-09-15T05:13:15.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:13:15.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDrbOZmZzfwGxAAZsdaHDiRu3TgwwzAoAiW3Q== 2024-09-15T05:13:15.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDrbOZmZzfwGxAAZsdaHDiRu3TgwwzAoAiW3Q=="}' 2024-09-15T05:13:15.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c5d6bdc9-c457-47c4-b073-b6c0ecf4327b -i td/osd-recovery-stats/5/new.json 2024-09-15T05:13:15.933 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:13:15.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-15T05:13:15.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDrbOZmZzfwGxAAZsdaHDiRu3TgwwzAoAiW3Q== --osd-uuid c5d6bdc9-c457-47c4-b073-b6c0ecf4327b 2024-09-15T05:13:15.969 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.968+0000 7fec7bc28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:15.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.973+0000 7fec7bc28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:15.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.974+0000 7fec7bc28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:15.975 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.974+0000 7fec7bc28440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:13:15.975 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.974+0000 7fec7bc28440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:13:15.975 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:15.974+0000 7fec7bc28440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:13:16.057 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:16.057+0000 7fec7bc28440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-15T05:13:22.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-15T05:13:22.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:13:22.217 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:13:22.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:13:22.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:13:22.700 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:13:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:13:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:13:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:13:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:13:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:13:22.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:22.722+0000 7f98f9c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:22.725 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:22.724+0000 7f98f9c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:22.726 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:22.725+0000 7f98f9c39440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:13:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:13:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:13:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:13:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:13:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:13:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:23.079 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:13:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:13:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:23.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:23.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:23.247+0000 7f98f9c39440 -1 Falling back to public interface 2024-09-15T05:13:23.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:24.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:24.429 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:13:24.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:24.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:13:24.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:24.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:24.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:25.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:25.798 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:13:25.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:25.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:13:25.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:25.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:25.814 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:13:25.813+0000 7f98f9c39440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:13:26.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:27.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:27.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:27.269 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:13:27.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:13:27.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:27.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:27.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:28.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:28.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:28.842 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:13:28.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:13:28.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:28.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:29.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:13:30.301 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:13:30.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:30.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:30.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:31.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:31.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:31.691 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:13:31.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:13:31.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:31.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:32.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:33.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:33.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:13:33.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:13:33.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:33.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:34.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:34.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:34.592 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:13:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:13:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:34.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:34.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:35.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:35.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:35.990 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:13:35.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:13:35.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:35.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:36.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:37.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:37.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:37.397 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:13:37.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:13:37.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:37.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:37.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:38.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:38.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:38.807 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:13:38.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:13:38.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:38.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:39.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:40.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:40.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:40.783 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:13:40.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:13:40.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:40.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:41.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:42.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:42.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:42.648 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:13:42.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:13:42.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:42.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:43.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:44.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:44.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:44.050 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:13:44.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:13:44.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:44.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:44.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:45.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:45.830 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:13:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:13:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:45.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:46.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:47.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:47.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:47.586 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:13:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:13:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:47.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:48.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:13:49.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:13:49.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:13:49.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:13:49.399 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:13:49.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:13:49.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:13:49.806 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/348969,v1:127.0.0.1:6843/348969] [v2:127.0.0.1:6844/348969,v1:127.0.0.1:6845/348969] exists,up c5d6bdc9-c457-47c4-b073-b6c0ecf4327b 2024-09-15T05:13:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:13:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:13:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:13:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-09-15T05:13:49.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-09-15T05:13:49.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-09-15T05:13:50.400 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:13:50.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:13:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-09-15T05:13:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:13:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:13:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:13:51.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:13:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:13:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:13:51.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:13:51.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:13:51.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:13:51.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:13:51.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:13:51.874 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:13:51.874 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:13:51.874 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:13:51.874 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:13:51.874 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:13:51.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:13:51.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:51.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:13:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-15T05:13:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-15T05:13:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-15T05:13:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:51.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:13:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-15T05:13:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-15T05:13:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996' 2024-09-15T05:13:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:52.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:13:52.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509465 2024-09-15T05:13:52.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509465 2024-09-15T05:13:52.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509465' 2024-09-15T05:13:52.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:52.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:13:52.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345942 2024-09-15T05:13:52.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345942 2024-09-15T05:13:52.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509465 3-85899345942' 2024-09-15T05:13:52.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:52.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:13:52.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182415 2024-09-15T05:13:52.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182415 2024-09-15T05:13:52.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509465 3-85899345942 4-107374182415' 2024-09-15T05:13:52.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:13:52.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018886 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018886 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672996 2-64424509465 3-85899345942 4-107374182415 5-128849018886' 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-15T05:13:52.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:52.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:13:52.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-15T05:13:52.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:52.623 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836521 2024-09-15T05:13:52.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-15T05:13:52.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-15T05:13:52.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:13:52.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-15T05:13:52.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:52.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-15T05:13:52.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:52.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:13:52.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-15T05:13:52.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:52.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-15T05:13:52.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-15T05:13:52.973 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672996 2024-09-15T05:13:52.973 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:13:53.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-15T05:13:53.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:53.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509465 2024-09-15T05:13:53.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:53.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:13:53.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509465 2024-09-15T05:13:53.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:53.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509465 2024-09-15T05:13:53.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-15T05:13:53.330 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509465 2024-09-15T05:13:53.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:13:53.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-15T05:13:53.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:53.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345942 2024-09-15T05:13:53.681 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:53.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:13:53.682 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345942 2024-09-15T05:13:53.683 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345942 2024-09-15T05:13:53.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-09-15T05:13:53.684 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345942 2024-09-15T05:13:53.684 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:13:54.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-15T05:13:54.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:54.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182415 2024-09-15T05:13:54.031 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:54.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:13:54.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182415 2024-09-15T05:13:54.033 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:54.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182415 2024-09-15T05:13:54.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-09-15T05:13:54.034 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182415 2024-09-15T05:13:54.034 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:13:54.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182413 -lt 107374182415 2024-09-15T05:13:54.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:13:55.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:13:55.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:13:55.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182415 -lt 107374182415 2024-09-15T05:13:55.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:13:55.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018886 2024-09-15T05:13:55.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:13:55.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:13:55.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018886 2024-09-15T05:13:55.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:13:55.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018886 2024-09-15T05:13:55.776 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018886 2024-09-15T05:13:55.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-09-15T05:13:55.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:13:56.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018886 -lt 128849018886 2024-09-15T05:13:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:13:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:13:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:13:56.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:13:57.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:13:57.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:13:57.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:13:57.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:13:57.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:13:57.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:13:57.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:13:57.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-09-15T05:13:57.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:57.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-09-15T05:13:57.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:57.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-09-15T05:13:57.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:57.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-09-15T05:13:57.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:57.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-09-15T05:13:58.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-09-15T05:13:58.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-09-15T05:13:58.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-09-15T05:13:58.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-09-15T05:13:58.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-09-15T05:13:58.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-09-15T05:13:58.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-09-15T05:13:58.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-09-15T05:13:58.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:58.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-09-15T05:13:59.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-09-15T05:13:59.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-09-15T05:13:59.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-09-15T05:13:59.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-09-15T05:13:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-09-15T05:13:59.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-09-15T05:13:59.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-09-15T05:13:59.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:13:59.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-09-15T05:14:00.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-09-15T05:14:00.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-09-15T05:14:00.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-09-15T05:14:00.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-09-15T05:14:00.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-09-15T05:14:00.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-09-15T05:14:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-09-15T05:14:00.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-09-15T05:14:00.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-09-15T05:14:00.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:00.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-09-15T05:14:01.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-09-15T05:14:01.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-09-15T05:14:01.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-09-15T05:14:01.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-09-15T05:14:01.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-09-15T05:14:01.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-09-15T05:14:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-09-15T05:14:01.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:01.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-09-15T05:14:02.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-09-15T05:14:02.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-09-15T05:14:02.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-09-15T05:14:02.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-09-15T05:14:02.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-09-15T05:14:02.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-09-15T05:14:02.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-09-15T05:14:02.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-09-15T05:14:02.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-09-15T05:14:02.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:02.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-09-15T05:14:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-09-15T05:14:03.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-09-15T05:14:03.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-09-15T05:14:03.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-09-15T05:14:03.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-09-15T05:14:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-09-15T05:14:03.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-09-15T05:14:03.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-09-15T05:14:03.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:03.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-09-15T05:14:04.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-09-15T05:14:04.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-09-15T05:14:04.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-09-15T05:14:04.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-09-15T05:14:04.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-09-15T05:14:04.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-09-15T05:14:04.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-09-15T05:14:04.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-09-15T05:14:04.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:04.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-09-15T05:14:05.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-09-15T05:14:05.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-09-15T05:14:05.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-09-15T05:14:05.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-09-15T05:14:05.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-09-15T05:14:05.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-09-15T05:14:05.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-09-15T05:14:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-09-15T05:14:05.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:05.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-09-15T05:14:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-09-15T05:14:06.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-09-15T05:14:06.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-09-15T05:14:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-09-15T05:14:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-09-15T05:14:06.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-09-15T05:14:06.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-09-15T05:14:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-09-15T05:14:07.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-09-15T05:14:07.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-09-15T05:14:07.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-09-15T05:14:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-09-15T05:14:07.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-09-15T05:14:07.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-09-15T05:14:07.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-09-15T05:14:07.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-09-15T05:14:07.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:07.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-09-15T05:14:08.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-09-15T05:14:08.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-09-15T05:14:08.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-09-15T05:14:08.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-09-15T05:14:08.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-09-15T05:14:08.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-09-15T05:14:08.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-09-15T05:14:08.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-09-15T05:14:08.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:08.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-09-15T05:14:09.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-09-15T05:14:09.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-09-15T05:14:09.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-09-15T05:14:09.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-09-15T05:14:09.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-09-15T05:14:09.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-09-15T05:14:09.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-09-15T05:14:09.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-09-15T05:14:09.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:09.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-09-15T05:14:10.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-09-15T05:14:10.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-09-15T05:14:10.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-09-15T05:14:10.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-09-15T05:14:10.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-09-15T05:14:10.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-09-15T05:14:10.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-09-15T05:14:10.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-09-15T05:14:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:10.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-09-15T05:14:11.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-09-15T05:14:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-09-15T05:14:11.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-09-15T05:14:11.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-09-15T05:14:11.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-09-15T05:14:11.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-09-15T05:14:11.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-09-15T05:14:11.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-09-15T05:14:11.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:11.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-09-15T05:14:12.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-09-15T05:14:12.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-09-15T05:14:12.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-09-15T05:14:12.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-09-15T05:14:12.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-09-15T05:14:12.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-09-15T05:14:12.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-09-15T05:14:12.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-09-15T05:14:12.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:12.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-09-15T05:14:13.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-09-15T05:14:13.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-09-15T05:14:13.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-09-15T05:14:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-09-15T05:14:13.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-09-15T05:14:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-09-15T05:14:13.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-09-15T05:14:13.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-09-15T05:14:13.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-09-15T05:14:13.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:13.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-09-15T05:14:14.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-09-15T05:14:14.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-09-15T05:14:14.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-09-15T05:14:14.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-09-15T05:14:14.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-09-15T05:14:14.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-09-15T05:14:14.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-09-15T05:14:14.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-09-15T05:14:14.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:14.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-09-15T05:14:15.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-09-15T05:14:15.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-09-15T05:14:15.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-09-15T05:14:15.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-09-15T05:14:15.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-09-15T05:14:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-09-15T05:14:15.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-09-15T05:14:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-09-15T05:14:15.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:15.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-09-15T05:14:16.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-09-15T05:14:16.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-09-15T05:14:16.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-09-15T05:14:16.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-09-15T05:14:16.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-09-15T05:14:16.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-09-15T05:14:16.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-09-15T05:14:16.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:16.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-09-15T05:14:17.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-09-15T05:14:17.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-09-15T05:14:17.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-09-15T05:14:17.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-09-15T05:14:17.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-09-15T05:14:17.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-09-15T05:14:17.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-09-15T05:14:17.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-09-15T05:14:17.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:17.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-09-15T05:14:18.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-09-15T05:14:18.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-09-15T05:14:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-09-15T05:14:18.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-09-15T05:14:18.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-09-15T05:14:18.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-09-15T05:14:18.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-09-15T05:14:18.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-09-15T05:14:18.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:18.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-09-15T05:14:19.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-09-15T05:14:19.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-09-15T05:14:19.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-09-15T05:14:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-09-15T05:14:19.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-09-15T05:14:19.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-09-15T05:14:19.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-09-15T05:14:19.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-09-15T05:14:19.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-09-15T05:14:19.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-09-15T05:14:19.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:14:19.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:14:19.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:14:19.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:14:20.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-09-15T05:14:20.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-09-15T05:14:20.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:14:20.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:14:20.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:14:20.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:14:20.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-09-15T05:14:20.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-09-15T05:14:20.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:14:20.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-15T05:14:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-15T05:14:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:14:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:14:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:14:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:14:21.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T05:14:21.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-15T05:14:21.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T05:14:21.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-09-15T05:14:21.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-09-15T05:14:22.220 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:14:22.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-09-15T05:14:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 340198 2024-09-15T05:14:22.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-09-15T05:14:22.628 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:14:22.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-09-15T05:14:23.478 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:14:23.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-09-15T05:14:24.900 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:14:24.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-09-15T05:14:24.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:14:24.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:14:24.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:14:24.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:14:25.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T05:14:25.449 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:14:25.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-09-15T05:14:27.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-09-15T05:14:27.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:14:27.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:14:27.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:14:27.460 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:14:27.461 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:14:27.461 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:14:27.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:14:27.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:14:27.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:14:27.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:14:27.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:14:27.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:27.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:14:28.011 INFO:tasks.workunit.client.0.smithi138.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-15T05:14:28.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq= 2024-09-15T05:14:28.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z '' 2024-09-15T05:14:28.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2176: flush_pg_stats: continue 2024-09-15T05:14:28.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:28.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:14:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673005 2024-09-15T05:14:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673005 2024-09-15T05:14:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005' 2024-09-15T05:14:28.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:28.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:14:28.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509475 2024-09-15T05:14:28.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509475 2024-09-15T05:14:28.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475' 2024-09-15T05:14:28.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:28.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:14:28.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-15T05:14:28.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-15T05:14:28.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-85899345951' 2024-09-15T05:14:28.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:28.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:14:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182424 2024-09-15T05:14:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182424 2024-09-15T05:14:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-85899345951 4-107374182424' 2024-09-15T05:14:28.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:14:28.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:14:28.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018895 2024-09-15T05:14:28.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018895 2024-09-15T05:14:28.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 1-42949673005 2-64424509475 3-85899345951 4-107374182424 5-128849018895' 2024-09-15T05:14:28.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:14:28.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673005 2024-09-15T05:14:28.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:14:28.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:14:28.602 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673005 2024-09-15T05:14:28.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:14:28.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673005 2024-09-15T05:14:28.604 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673005 2024-09-15T05:14:28.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-09-15T05:14:28.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:14:28.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673006 -lt 42949673005 2024-09-15T05:14:28.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:14:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509475 2024-09-15T05:14:28.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:14:28.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:14:28.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509475 2024-09-15T05:14:28.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:14:28.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509475 2024-09-15T05:14:28.959 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509475 2024-09-15T05:14:28.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-15T05:14:28.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:14:29.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-15T05:14:29.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:14:29.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-15T05:14:29.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:14:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:14:29.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-15T05:14:29.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:14:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-15T05:14:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-15T05:14:29.328 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345951 2024-09-15T05:14:29.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:14:29.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-15T05:14:29.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:14:29.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182424 2024-09-15T05:14:29.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:14:29.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:14:29.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182424 2024-09-15T05:14:29.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:14:29.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182424 2024-09-15T05:14:29.771 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182424 2024-09-15T05:14:29.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182424' 2024-09-15T05:14:29.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:14:30.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182422 -lt 107374182424 2024-09-15T05:14:30.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:14:31.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:14:31.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:14:31.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182425 -lt 107374182424 2024-09-15T05:14:31.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:14:31.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018895 2024-09-15T05:14:31.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:14:31.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:14:31.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018895 2024-09-15T05:14:31.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:14:31.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018895 2024-09-15T05:14:31.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018895' 2024-09-15T05:14:31.515 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018895 2024-09-15T05:14:31.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:14:31.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018896 -lt 128849018895 2024-09-15T05:14:31.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:14:31.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:31.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:32.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:14:32.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:14:32.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:14:32.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:14:32.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:14:32.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:14:32.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:14:32.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:14:33.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:14:33.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:14:33.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:33.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:14:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:14:33.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:14:34.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:14:34.049 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:14:34.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:34.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:14:34.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:14:34.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:14:35.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15 2024-09-15T05:14:35.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15 '!=' null 2024-09-15T05:14:35.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:14:35.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:14:35.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:14:35.359 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:14:35.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:14:35.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:14:35.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:35.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:36.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:14:36.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:14:36.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:14:36.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:14:36.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-15T05:14:36.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-15T05:14:36.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:14:36.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:14:36.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:14:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:14:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:14:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:14:37.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:14:37.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:14:37.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:37.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:14:37.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:14:38.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-09-15T05:14:38.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-09-15T05:14:38.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:14:38.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:14:38.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:14:38.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:14:38.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:14:38.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:14:38.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:14:38.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:14:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:14:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:14:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:14:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-15T05:14:39.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:14:39.236 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.236 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:14:39.236 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:14:39.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-09-15T05:14:39.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:14:39.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:14:39.247 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.247 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-15T05:14:39.248 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:14:39.248 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:14:39.248 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:14:39.254 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:14:39.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:14:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-09-15T05:14:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:14:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:14:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:14:39.268 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:14:39.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:14:39.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:14:39.848 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:14:39.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-09-15T05:14:39.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:14:39.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:14:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:14:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:14:39.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:14:45.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:14:45.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:14:45.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:14:45.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:14:45.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:14:45.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:14:45.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:14:45.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:14:45.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:14:45.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:14:45.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:14:45.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:14:45.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:14:45.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:14:45.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:14:45.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:14:45.228 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:14:45.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:14:45.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:14:45.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:14:45.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:14:45.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:14:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:14:45.234 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:14:45.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:14:45.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:14:45.235 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:14:45.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:14:45.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:14:45.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:14:45.236 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:14:45.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:14:45.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:14:45.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:14:45.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:14:45.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:14:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:14:45.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:14:45.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:14:45.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:14:45.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:14:45.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:14:45.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:14:45.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.620 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:14:45.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:14:45.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:14:45.873 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:14:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:14:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.874 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:14:45.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:14:45.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:14:45.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:14:45.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:14:45.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:14:45.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:14:46.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:14:46.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:14:46.867 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:14:46.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:14:46.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:14:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:14:46.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=10604fa1-2c84-444b-8ae8-1a3f361630ac 2024-09-15T05:14:46.897 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 10604fa1-2c84-444b-8ae8-1a3f361630ac 2024-09-15T05:14:46.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 10604fa1-2c84-444b-8ae8-1a3f361630ac' 2024-09-15T05:14:46.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:14:46.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGbeZm+lslNhAA2xbkJAEfsgbQocA+bAeahQ== 2024-09-15T05:14:46.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGbeZm+lslNhAA2xbkJAEfsgbQocA+bAeahQ=="}' 2024-09-15T05:14:46.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 10604fa1-2c84-444b-8ae8-1a3f361630ac -i td/osd-recovery-stats/0/new.json 2024-09-15T05:14:47.209 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:14:47.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:14:47.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGbeZm+lslNhAA2xbkJAEfsgbQocA+bAeahQ== --osd-uuid 10604fa1-2c84-444b-8ae8-1a3f361630ac 2024-09-15T05:14:47.243 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.243+0000 7fe5a5d1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:47.256 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.255+0000 7fe5a5d1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:47.257 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.256+0000 7fe5a5d1c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:47.257 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.257+0000 7fe5a5d1c440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:14:47.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.257+0000 7fe5a5d1c440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:14:47.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.257+0000 7fe5a5d1c440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:14:47.296 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:47.295+0000 7fe5a5d1c440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:14:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:14:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:14:52.651 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:14:52.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:14:52.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:14:53.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:14:53.135 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:14:53.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:14:53.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:14:53.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:14:53.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:14:53.157 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:53.156+0000 7fc126814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:53.159 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:53.159+0000 7fc126814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:53.160 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:53.160+0000 7fc126814440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:14:53.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:14:53.485 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:14:53.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:14:53.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:14:53.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:14:53.679 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:53.678+0000 7fc126814440 -1 Falling back to public interface 2024-09-15T05:14:53.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:14:54.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:14:54.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:14:54.852 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:14:54.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:14:54.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:14:54.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:14:55.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:14:56.154 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:14:56.153+0000 7fc126814440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:14:56.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:14:56.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:14:56.203 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:14:56.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:14:56.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:14:56.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:14:56.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:14:57.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:14:57.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:14:57.738 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:14:57.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:14:57.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:14:57.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:14:58.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:14:59.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:14:59.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:14:59.235 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:14:59.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:14:59.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:14:59.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:14:59.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:00.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:00.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:00.639 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:15:00.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:15:00.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:00.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:01.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:02.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:02.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:02.020 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:15:02.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:15:02.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:02.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:02.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:03.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:03.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:03.408 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:15:03.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:15:03.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:03.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:03.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:04.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:04.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:04.874 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:15:04.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:15:04.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:04.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:05.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:06.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:06.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:06.260 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:15:06.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:15:06.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:06.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:06.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:07.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:07.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:07.620 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:15:07.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:15:07.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:07.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:08.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:09.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:09.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:09.255 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:15:09.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:15:09.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:09.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:10.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:11.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:11.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:11.064 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:15:11.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:15:11.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:11.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:11.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:12.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:12.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:12.924 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:15:12.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:15:12.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:12.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:14.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:15.112 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:15:15.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:15:15.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:15.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:15.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:16.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:16.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:16.839 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:15:16.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:15:16.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:16.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:17.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:18.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:18.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:18.363 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:15:18.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:15:18.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:18.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:18.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:19.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:19.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:19.788 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:15:19.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:15:19.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:19.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:15:20.187 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/356609,v1:127.0.0.1:6803/356609] [v2:127.0.0.1:6804/356609,v1:127.0.0.1:6805/356609] exists,up 10604fa1-2c84-444b-8ae8-1a3f361630ac 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:15:20.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:15:20.192 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:15:20.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52c4b866-b279-46e0-827c-2f66dfae58d3 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 52c4b866-b279-46e0-827c-2f66dfae58d3 2024-09-15T05:15:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 52c4b866-b279-46e0-827c-2f66dfae58d3' 2024-09-15T05:15:20.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:15:20.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBobeZm9+kwDBAAJDL+DuIhUCXA0o9jRhK4dw== 2024-09-15T05:15:20.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBobeZm9+kwDBAAJDL+DuIhUCXA0o9jRhK4dw=="}' 2024-09-15T05:15:20.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52c4b866-b279-46e0-827c-2f66dfae58d3 -i td/osd-recovery-stats/1/new.json 2024-09-15T05:15:20.670 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:15:20.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:15:20.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBobeZm9+kwDBAAJDL+DuIhUCXA0o9jRhK4dw== --osd-uuid 52c4b866-b279-46e0-827c-2f66dfae58d3 2024-09-15T05:15:20.706 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.705+0000 7f110552b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:20.708 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.707+0000 7f110552b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:20.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.708+0000 7f110552b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.709+0000 7f110552b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:15:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.709+0000 7f110552b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:15:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.709+0000 7f110552b440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:15:20.785 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:20.785+0000 7f110552b440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:15:26.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:15:26.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:15:26.625 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:15:26.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:15:26.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:15:27.107 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:15:27.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:15:27.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:15:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:15:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:15:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:15:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:27.129+0000 7fcb7a032440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:27.131+0000 7fcb7a032440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:27.133 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:27.132+0000 7fcb7a032440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:15:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:27.504 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:15:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:15:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:27.654 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:27.654+0000 7fcb7a032440 -1 Falling back to public interface 2024-09-15T05:15:27.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:28.850 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:15:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:15:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:28.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:29.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:30.119 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:30.118+0000 7fcb7a032440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:15:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:30.209 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:15:30.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:15:30.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:30.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:30.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:31.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:31.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:31.771 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:15:31.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:15:31.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:31.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:32.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:33.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:33.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:33.211 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:15:33.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:15:33.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:33.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:33.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:34.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:34.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:34.644 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:15:34.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:15:34.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:34.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:35.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:36.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:36.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:36.081 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:15:36.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:15:36.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:36.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:36.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:37.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:37.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:37.462 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:15:37.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:15:37.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:37.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:38.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:15:38.872 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:15:38.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:38.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:39.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:40.343 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:15:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:15:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:40.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:40.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:41.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:41.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:41.933 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:15:41.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:15:41.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:41.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:42.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:43.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:43.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:43.498 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:15:43.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:15:43.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:43.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:44.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:45.393 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:15:45.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:15:45.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:45.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:46.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:47.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:47.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:47.246 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:15:47.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:15:47.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:47.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:48.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:49.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:49.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:49.015 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:15:49.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:15:49.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:49.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:50.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:51.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:51.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:51.204 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:15:51.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:15:51.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:51.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:51.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:15:52.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:15:52.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:15:52.984 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:15:52.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:15:52.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:15:52.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:15:53.427 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/358363,v1:127.0.0.1:6811/358363] [v2:127.0.0.1:6812/358363,v1:127.0.0.1:6813/358363] exists,up 52c4b866-b279-46e0-827c-2f66dfae58d3 2024-09-15T05:15:53.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:15:53.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:15:53.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:15:53.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:15:53.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ae681e72-ff1f-4c83-a20c-d187d60c0647 2024-09-15T05:15:53.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ae681e72-ff1f-4c83-a20c-d187d60c0647' 2024-09-15T05:15:53.434 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 ae681e72-ff1f-4c83-a20c-d187d60c0647 2024-09-15T05:15:53.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:15:53.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJbeZmU+53GhAA3Y3YFpMkII9zxdah7my5rQ== 2024-09-15T05:15:53.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJbeZmU+53GhAA3Y3YFpMkII9zxdah7my5rQ=="}' 2024-09-15T05:15:53.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ae681e72-ff1f-4c83-a20c-d187d60c0647 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:15:53.912 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:15:53.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:15:53.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJbeZmU+53GhAA3Y3YFpMkII9zxdah7my5rQ== --osd-uuid ae681e72-ff1f-4c83-a20c-d187d60c0647 2024-09-15T05:15:53.948 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.945+0000 7f8361a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.947+0000 7f8361a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.948+0000 7f8361a75440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:15:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.949+0000 7f8361a75440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:15:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.949+0000 7f8361a75440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:15:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:53.949+0000 7f8361a75440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:15:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:15:54.043+0000 7f8361a75440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:16:00.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:16:00.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:16:00.199 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:16:00.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:16:00.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:16:00.685 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:16:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:16:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:16:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:16:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:16:00.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:16:00.708 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:00.707+0000 7fa04504f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:00.712 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:00.709+0000 7fa04504f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:00.712 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:00.710+0000 7fa04504f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:01.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:16:01.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:16:01.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:16:01.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:01.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:01.227 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:01.227+0000 7fa04504f440 -1 Falling back to public interface 2024-09-15T05:16:01.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:02.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:02.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:02.413 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:16:02.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:16:02.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:02.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:02.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:03.747 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:03.746+0000 7fa04504f440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:16:03.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:03.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:03.769 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:16:03.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:16:03.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:03.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:04.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:05.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:05.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:05.417 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:16:05.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:16:05.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:05.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:05.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:06.856 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:16:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:16:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:06.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:07.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:08.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:08.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:08.330 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:16:08.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:16:08.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:08.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:08.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:09.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:09.782 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:16:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:16:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:10.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:11.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:11.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:11.184 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:16:11.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:16:11.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:11.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:11.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:12.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:12.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:12.641 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:16:12.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:16:12.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:12.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:13.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:14.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:14.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:14.024 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:16:14.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:16:14.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:14.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:14.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:15.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:15.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:15.530 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:16:15.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:16:15.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:15.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:16.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:17.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:17.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:17.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:16:17.043 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:16:17.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:17.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:17.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:18.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:18.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:18.796 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:16:18.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:16:18.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:18.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:19.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:20.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:20.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:20.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:16:20.657 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:16:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:20.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:21.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:22.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:22.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:22.880 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:16:22.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:16:22.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:22.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:23.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:24.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:24.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:24.296 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:16:24.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:16:24.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:24.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:24.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:25.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:25.853 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:16:25.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:16:25.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:25.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:26.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:27.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:27.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:27.581 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:16:27.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:16:27.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:27.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:16:27.933 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/360092,v1:127.0.0.1:6819/360092] [v2:127.0.0.1:6820/360092,v1:127.0.0.1:6821/360092] exists,up ae681e72-ff1f-4c83-a20c-d187d60c0647 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:16:27.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:16:27.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:16:27.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:16:27.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=435b9129-137d-434b-b6c5-6fb46789031f 2024-09-15T05:16:27.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 435b9129-137d-434b-b6c5-6fb46789031f' 2024-09-15T05:16:27.941 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 435b9129-137d-434b-b6c5-6fb46789031f 2024-09-15T05:16:27.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:16:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrbeZmPQueOBAA1iqNFjLrWXW6qzMtZWIr2w== 2024-09-15T05:16:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrbeZmPQueOBAA1iqNFjLrWXW6qzMtZWIr2w=="}' 2024-09-15T05:16:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 435b9129-137d-434b-b6c5-6fb46789031f -i td/osd-recovery-stats/3/new.json 2024-09-15T05:16:28.425 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:16:28.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T05:16:28.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrbeZmPQueOBAA1iqNFjLrWXW6qzMtZWIr2w== --osd-uuid 435b9129-137d-434b-b6c5-6fb46789031f 2024-09-15T05:16:28.461 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.460+0000 7fd1d6e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:28.462 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.462+0000 7fd1d6e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:28.463 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.463+0000 7fd1d6e17440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:28.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.463+0000 7fd1d6e17440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:16:28.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.463+0000 7fd1d6e17440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:16:28.464 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.463+0000 7fd1d6e17440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:16:28.549 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:28.548+0000 7fd1d6e17440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T05:16:34.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T05:16:34.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:16:34.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:16:34.480 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:16:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:16:34.965 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:16:34.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:16:34.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:16:34.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:16:34.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:16:34.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:16:34.988 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:34.987+0000 7f9087a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:34.989 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:34.989+0000 7f9087a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:34.990 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:34.990+0000 7f9087a0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:16:35.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:16:35.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:16:35.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:16:35.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:16:35.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:16:35.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:35.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:16:35.318 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:16:35.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:35.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:35.505 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:35.504+0000 7f9087a0b440 -1 Falling back to public interface 2024-09-15T05:16:35.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:36.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:36.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:36.671 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:16:36.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:16:36.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:36.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:37.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:37.986 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:16:37.985+0000 7f9087a0b440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:16:38.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:38.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:38.066 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:16:38.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:16:38.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:38.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:38.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:39.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:39.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:39.544 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:16:39.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:16:39.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:39.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:40.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:41.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:41.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:41.027 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:16:41.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:16:41.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:41.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:41.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:42.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:42.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:42.488 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:16:42.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:16:42.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:42.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:42.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:43.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:43.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:43.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:16:43.991 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:16:43.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:43.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:44.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:45.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:45.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:45.363 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:16:45.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:16:45.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:45.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:45.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:46.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:46.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:46.862 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:16:46.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:16:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:46.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:47.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:48.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:48.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:48.311 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:16:48.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:16:48.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:48.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:48.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:49.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:49.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:49.847 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:16:49.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:16:49.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:49.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:50.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:51.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:51.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:51.326 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:16:51.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:16:51.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:51.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:52.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:53.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:53.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:53.088 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:16:53.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:16:53.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:53.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:53.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:54.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:54.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:54.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:16:54.745 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:16:54.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:54.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:55.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:56.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:57.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:57.000 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:16:57.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:16:57.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:57.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:57.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:16:58.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:16:58.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:16:58.760 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:16:58.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:16:58.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:16:58.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:16:59.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:00.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:00.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:00.529 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:17:00.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:17:00.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:00.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:17:01.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:02.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:02.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:02.278 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:17:02.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:17:02.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:02.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:17:02.658 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/361846,v1:127.0.0.1:6827/361846] [v2:127.0.0.1:6828/361846,v1:127.0.0.1:6829/361846] exists,up 435b9129-137d-434b-b6c5-6fb46789031f 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-15T05:17:02.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:17:02.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:17:02.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e21ba3dd-6868-4da7-93db-759713a17cc7 2024-09-15T05:17:02.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 e21ba3dd-6868-4da7-93db-759713a17cc7' 2024-09-15T05:17:02.667 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 e21ba3dd-6868-4da7-93db-759713a17cc7 2024-09-15T05:17:02.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:17:02.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDObeZmE/BGKBAABAiHCLU7SjgMwLnwHUE7Qg== 2024-09-15T05:17:02.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDObeZmE/BGKBAABAiHCLU7SjgMwLnwHUE7Qg=="}' 2024-09-15T05:17:02.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e21ba3dd-6868-4da7-93db-759713a17cc7 -i td/osd-recovery-stats/4/new.json 2024-09-15T05:17:03.237 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:17:03.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-15T05:17:03.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDObeZmE/BGKBAABAiHCLU7SjgMwLnwHUE7Qg== --osd-uuid e21ba3dd-6868-4da7-93db-759713a17cc7 2024-09-15T05:17:03.273 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.272+0000 7fcc5c44d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:03.275 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.274+0000 7fcc5c44d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:03.276 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.275+0000 7fcc5c44d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:03.276 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.275+0000 7fcc5c44d440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:17:03.276 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.275+0000 7fcc5c44d440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:17:03.276 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.275+0000 7fcc5c44d440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:17:03.352 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:03.352+0000 7fcc5c44d440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-15T05:17:08.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-15T05:17:08.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:17:08.945 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T05:17:08.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T05:17:08.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:17:09.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T05:17:09.449 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T05:17:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:17:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:17:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:17:09.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:17:09.472 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:09.472+0000 7f72e4e14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:09.474 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:09.474+0000 7f72e4e14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:09.475 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:09.475+0000 7f72e4e14440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:09.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T05:17:09.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:17:09.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T05:17:09.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:17:09.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:17:09.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:09.800 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:17:09.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:17:09.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:09.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:09.995 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:09.994+0000 7f72e4e14440 -1 Falling back to public interface 2024-09-15T05:17:10.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:11.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:11.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:11.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:17:11.153 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:17:11.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:11.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:11.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:12.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:12.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:12.508 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:17:12.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:17:12.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:12.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:12.552 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:12.551+0000 7f72e4e14440 -1 osd.4 0 log_to_monitors true 2024-09-15T05:17:12.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:13.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:13.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:13.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:17:13.885 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:17:13.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:13.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:14.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:15.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:15.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:15.252 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:17:15.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:17:15.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:15.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:15.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:16.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:16.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:16.678 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:17:16.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:17:16.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:16.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:17.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:18.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:18.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:18.064 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:17:18.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:17:18.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:18.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:18.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:19.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:19.473 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:17:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:17:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:19.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:19.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:20.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:20.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:20.960 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:17:20.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:17:20.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:20.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:21.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:22.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:22.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:22.412 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:17:22.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:17:22.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:22.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:22.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:23.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:23.852 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:17:23.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:17:23.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:23.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:24.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:25.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:25.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:25.234 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:17:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:17:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:25.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:27.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:27.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:27.103 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:17:27.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:17:27.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:27.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:28.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:29.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:29.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:29.235 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:17:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:17:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:29.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:29.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:30.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:30.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:30.963 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:17:30.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:17:30.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:30.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:31.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:32.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:32.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:32.666 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:17:32.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:17:32.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:32.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:33.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:34.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:34.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:34.406 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:17:34.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:17:34.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:34.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:35.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:36.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:36.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:36.025 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:17:36.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:17:36.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:36.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:36.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:37.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:37.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:37.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:17:37.569 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:17:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:37.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:17:37.922 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/363601,v1:127.0.0.1:6835/363601] [v2:127.0.0.1:6836/363601,v1:127.0.0.1:6837/363601] exists,up e21ba3dd-6868-4da7-93db-759713a17cc7 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:17:37.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:17:37.927 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-15T05:17:37.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:17:37.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=65783d56-50d0-48a7-93d4-294f9f684e53 2024-09-15T05:17:37.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 65783d56-50d0-48a7-93d4-294f9f684e53' 2024-09-15T05:17:37.929 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 65783d56-50d0-48a7-93d4-294f9f684e53 2024-09-15T05:17:37.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:17:37.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxbeZmFoAAOBAAs0Zo4MJHYEfju53TLOjzXw== 2024-09-15T05:17:37.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxbeZmFoAAOBAAs0Zo4MJHYEfju53TLOjzXw=="}' 2024-09-15T05:17:37.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 65783d56-50d0-48a7-93d4-294f9f684e53 -i td/osd-recovery-stats/5/new.json 2024-09-15T05:17:38.464 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:17:38.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-15T05:17:38.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxbeZmFoAAOBAAs0Zo4MJHYEfju53TLOjzXw== --osd-uuid 65783d56-50d0-48a7-93d4-294f9f684e53 2024-09-15T05:17:38.500 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.500+0000 7f742e762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:38.521 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.521+0000 7f742e762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:38.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.522+0000 7f742e762440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:38.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.522+0000 7f742e762440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:17:38.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.522+0000 7f742e762440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:17:38.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.522+0000 7f742e762440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:17:38.607 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:38.606+0000 7f742e762440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-15T05:17:44.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-15T05:17:44.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:17:44.454 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:17:44.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:17:44.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:17:44.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:17:44.947 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:17:44.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:17:44.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:17:44.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:17:44.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:17:44.969 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:44.968+0000 7ff78c208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:44.971 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:44.970+0000 7ff78c208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:44.972 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:44.971+0000 7ff78c208440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:17:45.299 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:17:45.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:45.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:45.491 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:45.490+0000 7ff78c208440 -1 Falling back to public interface 2024-09-15T05:17:45.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:46.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:46.655 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:17:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:17:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:46.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:47.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:47.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:17:47.972+0000 7ff78c208440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:17:48.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:48.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:48.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:17:48.015 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:17:48.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:48.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:48.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:49.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:49.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:49.567 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:17:49.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:17:49.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:49.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:50.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:51.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:51.030 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:17:51.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:51.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:17:51.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:51.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:51.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:52.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:52.566 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:17:52.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:17:52.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:52.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:53.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:54.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:54.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:54.099 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:17:54.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:17:54.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:54.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:54.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:55.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:55.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:55.493 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:17:55.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:17:55.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:55.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:55.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:56.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:17:56.952 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:17:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:57.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:58.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:58.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:58.387 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:17:58.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:17:58.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:58.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:17:58.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:17:59.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:17:59.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:17:59.903 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:17:59.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:17:59.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:17:59.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:00.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:01.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:01.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:01.286 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:18:01.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:18:01.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:01.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:02.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:03.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:18:03.007 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:18:03.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:03.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:03.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:04.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:04.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:04.736 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:18:04.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:18:04.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:04.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:05.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:06.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:06.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:06.817 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:18:06.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:18:06.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:06.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:07.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:08.570 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:18:08.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:18:08.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:08.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:09.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:10.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:10.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:10.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:18:10.322 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:18:10.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:10.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:11.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:18:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:18:12.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:18:12.165 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:18:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:18:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:18:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:18:12.579 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/365380,v1:127.0.0.1:6843/365380] [v2:127.0.0.1:6844/365380,v1:127.0.0.1:6845/365380] exists,up 65783d56-50d0-48a7-93d4-294f9f684e53 2024-09-15T05:18:12.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:18:12.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:18:12.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:18:12.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-09-15T05:18:12.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T05:18:13.078 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:18:13.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:18:14.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-09-15T05:18:14.946 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 4 2024-09-15T05:18:14.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-09-15T05:18:14.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:18:14.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:18:14.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:18:14.959 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:18:14.959 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:18:14.959 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:18:14.959 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:18:14.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:18:14.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:18:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:18:15.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:18:15.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:18:15.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:18:15.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:18:15.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:18:15.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:15.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:18:15.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-15T05:18:15.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-15T05:18:15.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T05:18:15.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:15.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:18:15.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-15T05:18:15.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-15T05:18:15.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-15T05:18:15.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:15.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:18:16.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509465 2024-09-15T05:18:16.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509465 2024-09-15T05:18:16.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509465' 2024-09-15T05:18:16.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:16.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:18:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345940 2024-09-15T05:18:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345940 2024-09-15T05:18:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509465 3-85899345940' 2024-09-15T05:18:16.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:16.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:18:16.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-15T05:18:16.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-15T05:18:16.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509465 3-85899345940 4-107374182413' 2024-09-15T05:18:16.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:16.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:18:16.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-15T05:18:16.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-15T05:18:16.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509465 3-85899345940 4-107374182413 5-128849018887' 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T05:18:16.323 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836524 2024-09-15T05:18:16.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:18:16.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-15T05:18:16.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:16.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-15T05:18:16.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:16.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:18:16.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-15T05:18:16.709 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:16.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-15T05:18:16.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-15T05:18:16.710 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672996 2024-09-15T05:18:16.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:18:17.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-15T05:18:17.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:17.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509465 2024-09-15T05:18:17.084 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:17.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:18:17.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509465 2024-09-15T05:18:17.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:17.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509465 2024-09-15T05:18:17.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-15T05:18:17.087 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509465 2024-09-15T05:18:17.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:18:17.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-15T05:18:17.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:17.439 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345940 2024-09-15T05:18:17.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:17.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:18:17.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345940 2024-09-15T05:18:17.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:17.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345940 2024-09-15T05:18:17.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345940' 2024-09-15T05:18:17.442 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345940 2024-09-15T05:18:17.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:18:17.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345940 -lt 85899345940 2024-09-15T05:18:17.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:17.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182413 2024-09-15T05:18:17.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:17.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:18:17.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182413 2024-09-15T05:18:17.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:17.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-15T05:18:17.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-09-15T05:18:17.806 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182413 2024-09-15T05:18:17.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:18:18.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182413 -lt 107374182413 2024-09-15T05:18:18.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:18.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018887 2024-09-15T05:18:18.196 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:18.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:18:18.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018887 2024-09-15T05:18:18.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-15T05:18:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-15T05:18:18.199 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018887 2024-09-15T05:18:18.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:18:18.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018888 -lt 128849018887 2024-09-15T05:18:18.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:18:18.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:18:18.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:18:19.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:18:19.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:18:19.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:18:19.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:18:19.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:18:19.401 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:18:19.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:18:19.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:18:19.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:18:19.869 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-09-15T05:18:19.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:19.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-09-15T05:18:20.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-09-15T05:18:20.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-09-15T05:18:20.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-09-15T05:18:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-09-15T05:18:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-09-15T05:18:20.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-09-15T05:18:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-09-15T05:18:20.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:20.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-09-15T05:18:21.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-09-15T05:18:21.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-09-15T05:18:21.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-09-15T05:18:21.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-09-15T05:18:21.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-09-15T05:18:21.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-09-15T05:18:21.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:21.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-09-15T05:18:22.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-09-15T05:18:22.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-09-15T05:18:22.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-09-15T05:18:22.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-09-15T05:18:22.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-09-15T05:18:22.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-09-15T05:18:22.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-09-15T05:18:22.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:22.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-09-15T05:18:23.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-09-15T05:18:23.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-09-15T05:18:23.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-09-15T05:18:23.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-09-15T05:18:23.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-09-15T05:18:23.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-09-15T05:18:23.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:23.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-09-15T05:18:24.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-09-15T05:18:24.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-09-15T05:18:24.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-09-15T05:18:24.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-09-15T05:18:24.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-09-15T05:18:24.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-09-15T05:18:24.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-09-15T05:18:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:24.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-09-15T05:18:25.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-09-15T05:18:25.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-09-15T05:18:25.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-09-15T05:18:25.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-09-15T05:18:25.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-09-15T05:18:25.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-09-15T05:18:25.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-09-15T05:18:25.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:25.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-09-15T05:18:26.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-09-15T05:18:26.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-09-15T05:18:26.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-09-15T05:18:26.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-09-15T05:18:26.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-09-15T05:18:26.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-09-15T05:18:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-09-15T05:18:26.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:26.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-09-15T05:18:27.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-09-15T05:18:27.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-09-15T05:18:27.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-09-15T05:18:27.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-09-15T05:18:27.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-09-15T05:18:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-09-15T05:18:27.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-09-15T05:18:27.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:27.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-09-15T05:18:28.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-09-15T05:18:28.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-09-15T05:18:28.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-09-15T05:18:28.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-09-15T05:18:28.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-09-15T05:18:28.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-09-15T05:18:28.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:28.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-09-15T05:18:29.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-09-15T05:18:29.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-09-15T05:18:29.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-09-15T05:18:29.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-09-15T05:18:29.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-09-15T05:18:29.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-09-15T05:18:29.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-09-15T05:18:29.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:29.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-09-15T05:18:30.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-09-15T05:18:30.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-09-15T05:18:30.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-09-15T05:18:30.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-09-15T05:18:30.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-09-15T05:18:30.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-09-15T05:18:30.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-09-15T05:18:30.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:30.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-09-15T05:18:31.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-09-15T05:18:31.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-09-15T05:18:31.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-09-15T05:18:31.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-09-15T05:18:31.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-09-15T05:18:31.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-09-15T05:18:31.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-09-15T05:18:31.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:31.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-09-15T05:18:32.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-09-15T05:18:32.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-09-15T05:18:32.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-09-15T05:18:32.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-09-15T05:18:32.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-09-15T05:18:32.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-09-15T05:18:32.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-09-15T05:18:32.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:32.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-09-15T05:18:33.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-09-15T05:18:33.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-09-15T05:18:33.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-09-15T05:18:33.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-09-15T05:18:33.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-09-15T05:18:33.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-09-15T05:18:33.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-09-15T05:18:33.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:33.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-09-15T05:18:34.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-09-15T05:18:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-09-15T05:18:34.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-09-15T05:18:34.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-09-15T05:18:34.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-09-15T05:18:34.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-09-15T05:18:34.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-09-15T05:18:34.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:34.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-09-15T05:18:35.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-09-15T05:18:35.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-09-15T05:18:35.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-09-15T05:18:35.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-09-15T05:18:35.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-09-15T05:18:35.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-09-15T05:18:35.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:35.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-09-15T05:18:36.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-09-15T05:18:36.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-09-15T05:18:36.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-09-15T05:18:36.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-09-15T05:18:36.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-09-15T05:18:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-09-15T05:18:36.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-09-15T05:18:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:36.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-09-15T05:18:37.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-09-15T05:18:37.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-09-15T05:18:37.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-09-15T05:18:37.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-09-15T05:18:37.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-09-15T05:18:37.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-09-15T05:18:37.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-09-15T05:18:37.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:37.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-09-15T05:18:38.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-09-15T05:18:38.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-09-15T05:18:38.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-09-15T05:18:38.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-09-15T05:18:38.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-09-15T05:18:38.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-09-15T05:18:38.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-09-15T05:18:38.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:38.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-09-15T05:18:39.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-09-15T05:18:39.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-09-15T05:18:39.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-09-15T05:18:39.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-09-15T05:18:39.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-09-15T05:18:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-09-15T05:18:39.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-09-15T05:18:39.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:39.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-09-15T05:18:40.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-09-15T05:18:40.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-09-15T05:18:40.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-09-15T05:18:40.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-09-15T05:18:40.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-09-15T05:18:40.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-09-15T05:18:40.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-09-15T05:18:40.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:40.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-09-15T05:18:41.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-09-15T05:18:41.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-09-15T05:18:41.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-09-15T05:18:41.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-09-15T05:18:41.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-09-15T05:18:41.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-09-15T05:18:41.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:41.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-09-15T05:18:42.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-09-15T05:18:42.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-09-15T05:18:42.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-09-15T05:18:42.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-09-15T05:18:42.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-09-15T05:18:42.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-09-15T05:18:42.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-09-15T05:18:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-09-15T05:18:43.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-09-15T05:18:43.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-09-15T05:18:43.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-09-15T05:18:43.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-09-15T05:18:43.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-09-15T05:18:43.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-09-15T05:18:43.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-09-15T05:18:43.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:43.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-09-15T05:18:44.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-09-15T05:18:44.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-09-15T05:18:44.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-09-15T05:18:44.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-09-15T05:18:44.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-09-15T05:18:44.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-09-15T05:18:44.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:44.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-09-15T05:18:45.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:45.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-09-15T05:18:45.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:45.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-09-15T05:18:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-09-15T05:18:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:45.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-09-15T05:18:45.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-09-15T05:18:45.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-09-15T05:18:45.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-09-15T05:18:45.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:18:45.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:18:45.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:18:45.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:18:45.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:18:46.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-09-15T05:18:46.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-09-15T05:18:46.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-09-15T05:18:46.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-09-15T05:18:46.321 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-09-15T05:18:46.321 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:18:46.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:0 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:4' 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-09-15T05:18:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-09-15T05:18:47.309 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:18:47.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-15T05:18:47.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-09-15T05:18:48.342 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already out. 2024-09-15T05:18:48.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-15T05:18:48.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-09-15T05:18:49.476 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:18:49.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-15T05:18:49.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-09-15T05:18:50.686 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already out. 2024-09-15T05:18:50.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-09-15T05:18:50.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-09-15T05:18:51.352 INFO:tasks.workunit.client.0.smithi138.stderr:osd.4 is already out. 2024-09-15T05:18:51.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-09-15T05:18:52.938 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T05:18:52.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-09-15T05:18:54.115 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:18:54.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-09-15T05:18:54.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:18:54.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:18:54.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:18:54.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:18:54.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-09-15T05:18:54.923 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:18:54.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-09-15T05:18:56.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-09-15T05:18:56.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:18:56.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:18:56.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:18:56.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:18:57.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:18:57.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:18:57.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:18:57.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:18:57.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:18:57.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:18:57.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:18:57.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:57.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:18:57.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836534 2024-09-15T05:18:57.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836534 2024-09-15T05:18:57.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534' 2024-09-15T05:18:57.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:57.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:18:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673006 2024-09-15T05:18:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673006 2024-09-15T05:18:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673006' 2024-09-15T05:18:57.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:57.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:18:57.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509475 2024-09-15T05:18:57.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509475 2024-09-15T05:18:57.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673006 2-64424509475' 2024-09-15T05:18:57.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:57.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:18:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-15T05:18:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-15T05:18:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673006 2-64424509475 3-85899345951' 2024-09-15T05:18:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:57.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:18:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182424 2024-09-15T05:18:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182424 2024-09-15T05:18:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673006 2-64424509475 3-85899345951 4-107374182424' 2024-09-15T05:18:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:18:58.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:18:58.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018898 2024-09-15T05:18:58.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018898 2024-09-15T05:18:58.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836534 1-42949673006 2-64424509475 3-85899345951 4-107374182424 5-128849018898' 2024-09-15T05:18:58.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:58.157 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836534 2024-09-15T05:18:58.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:58.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:18:58.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836534 2024-09-15T05:18:58.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:58.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836534 2024-09-15T05:18:58.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836534' 2024-09-15T05:18:58.160 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836534 2024-09-15T05:18:58.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:18:58.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836532 -lt 21474836534 2024-09-15T05:18:58.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:18:59.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:18:59.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:18:59.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836534 -lt 21474836534 2024-09-15T05:18:59.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:18:59.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673006 2024-09-15T05:18:59.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:18:59.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:18:59.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673006 2024-09-15T05:18:59.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:18:59.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673006 2024-09-15T05:18:59.903 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673006 2024-09-15T05:18:59.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673006' 2024-09-15T05:18:59.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:19:00.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673006 -lt 42949673006 2024-09-15T05:19:00.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:19:00.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509475 2024-09-15T05:19:00.480 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:19:00.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:19:00.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509475 2024-09-15T05:19:00.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:19:00.482 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509475 2024-09-15T05:19:00.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509475 2024-09-15T05:19:00.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-15T05:19:00.482 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:19:00.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509476 -lt 64424509475 2024-09-15T05:19:00.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:19:00.839 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345951 2024-09-15T05:19:00.839 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:19:00.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:19:00.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345951 2024-09-15T05:19:00.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:19:00.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-15T05:19:00.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-09-15T05:19:00.842 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345951 2024-09-15T05:19:00.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:19:01.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-15T05:19:01.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:19:01.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182424 2024-09-15T05:19:01.265 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:19:01.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182424 2024-09-15T05:19:01.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:19:01.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182424 2024-09-15T05:19:01.268 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182424 2024-09-15T05:19:01.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182424' 2024-09-15T05:19:01.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:19:01.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182424 -lt 107374182424 2024-09-15T05:19:01.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:19:01.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018898 2024-09-15T05:19:01.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:19:01.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:19:01.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018898 2024-09-15T05:19:01.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:19:01.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018898 2024-09-15T05:19:01.655 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018898 2024-09-15T05:19:01.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2024-09-15T05:19:01.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:19:02.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018898 -lt 128849018898 2024-09-15T05:19:02.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:19:02.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:02.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:02.646 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:02.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:02.647 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:03.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:03.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:03.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:03.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:03.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:04.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:04.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:04.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:04.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:04.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:04.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:04.603 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:19:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:19:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:05.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:05.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:05.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:05.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:05.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:05.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:05.635 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:06.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:06.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:06.108 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:19:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:19:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:06.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:07.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:07.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:07.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:07.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:07.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:07.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:07.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:07.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:08.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:19:08.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:19:08.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:08.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:08.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:08.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:08.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:08.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:08.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:08.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:08.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:08.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:08.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:08.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:08.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:08.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:08.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:09.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:09.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:09.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:09.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:09.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:09.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:09.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:09.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:09.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-09-15T05:19:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-09-15T05:19:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:09.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:09.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:09.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:09.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:09.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:09.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:09.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:09.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:09.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:10.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:10.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:10.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:10.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:10.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:10.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:10.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:10.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:10.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:10.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:10.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:10.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:10.617 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:11.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:19:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:19:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:11.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:11.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:11.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:11.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:11.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:11.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:12.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:12.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:19:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:19:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:12.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:12.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:12.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:12.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:12.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:12.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:12.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:12.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:13.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:13.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:13.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:13.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:13.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:13.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:13.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:14.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:19:14.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:19:14.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:14.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:14.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:14.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:14.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:14.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:14.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:14.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:14.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:14.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:14.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:14.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:14.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:14.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:14.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:15.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:15.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:15.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-09-15T05:19:15.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-09-15T05:19:15.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:15.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:15.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:15.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:15.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:16.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:16.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:16.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:16.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:16.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:19:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:19:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:17.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:17.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:17.489 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:17.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:17.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:17.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:17.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:17.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:17.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:17.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:18.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:18.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:18.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:18.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:18.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:18.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:19.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-09-15T05:19:19.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-09-15T05:19:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:19.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:19.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:19.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:19.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:19.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:19.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:19.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:20.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:20.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:20.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-15T05:19:20.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-15T05:19:20.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:20.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:20.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:20.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:20.770 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:21.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:21.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:21.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:21.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:21.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:21.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:21.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:21.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:21.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:22.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-09-15T05:19:22.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-09-15T05:19:22.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:22.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:22.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:22.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:22.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:22.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:22.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:22.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:22.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:23.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:23.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:23.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:23.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:19:23.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:19:23.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:23.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:23.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:23.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:23.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:23.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:23.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:23.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:23.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:23.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:23.578 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:23.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:19:23.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:23.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:23.913 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:19:24.409 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:19:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:19:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:19:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:19:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:19:24.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:19:24.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:19:24.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:19:24.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:19:24.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:19:24.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:19:24.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:19:24.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:19:24.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:19:25.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:19:25.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:19:25.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:19:25.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:19:25.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:19:25.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:19:26.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-09-15T05:19:26.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-09-15T05:19:26.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-09-15T05:19:26.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-09-15T05:19:26.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:19:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:19:26.188 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:19:26.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-09-15T05:19:26.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:19:26.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:19:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-09-15T05:19:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:19:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:19:26.203 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-09-15T05:19:26.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:19:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:19:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:19:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:19:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:19:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:19:26.215 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.218 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:19:26.219 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-09-15T05:19:26.226 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:19:26.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-09-15T05:19:26.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-09-15T05:19:26.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-09-15T05:19:26.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-09-15T05:19:26.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:19:26.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:19:26.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:19:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:19:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-09-15T05:19:26.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:19:26.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:19:26.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:19:26.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:19:26.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:19:32.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:19:32.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:19:32.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:19:32.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:19:32.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:19:32.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:19:32.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:19:32.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:19:32.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:19:32.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:19:32.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:19:32.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:19:32.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:19:32.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:19:32.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:19:32.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:19:32.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:19:32.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:19:32.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:19:32.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:19:32.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:19:32.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:19:32.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:19:32.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:19:32.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:19:32.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:19:32.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:19:32.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:19:32.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:19:32.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:19:32.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:19:32.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:19:32.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:19:32.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:19:32.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:19:32.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:19:32.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:19:32.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:19:32.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:19:32.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:19:32.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:19:32.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:19:32.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:19:32.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:19:32.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:19:32.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:19:32.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:19:32.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:19:32.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:19:32.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:19:32.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:19:32.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:19:32.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:19:32.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:19:32.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:19:32.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:19:32.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:19:32.732 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:19:32.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:19:32.815 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:32.816 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:32.816 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:19:32.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:19:32.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:19:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:19:32.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:19:33.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:19:33.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:19:33.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:19:33.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:19:33.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:19:33.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:19:33.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=52173352-e507-40bc-a6b6-970a77fb31ea 2024-09-15T05:19:33.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 52173352-e507-40bc-a6b6-970a77fb31ea' 2024-09-15T05:19:33.759 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 52173352-e507-40bc-a6b6-970a77fb31ea 2024-09-15T05:19:33.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:19:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlbuZmV5mwLRAAG1SuYZZpHHkZ01SYOJaXZg== 2024-09-15T05:19:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlbuZmV5mwLRAAG1SuYZZpHHkZ01SYOJaXZg=="}' 2024-09-15T05:19:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 52173352-e507-40bc-a6b6-970a77fb31ea -i td/osd-recovery-stats/0/new.json 2024-09-15T05:19:34.035 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:19:34.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:19:34.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlbuZmV5mwLRAAG1SuYZZpHHkZ01SYOJaXZg== --osd-uuid 52173352-e507-40bc-a6b6-970a77fb31ea 2024-09-15T05:19:34.070 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.069+0000 7f2ba9616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:34.082 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.081+0000 7f2ba9616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:34.083 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.082+0000 7f2ba9616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:34.083 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.082+0000 7f2ba9616440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:19:34.083 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.082+0000 7f2ba9616440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:19:34.083 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.082+0000 7f2ba9616440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:19:34.122 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:34.121+0000 7f2ba9616440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:19:39.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:19:39.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:19:39.701 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:19:39.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:19:39.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:19:40.183 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:19:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:19:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:19:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:19:40.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:19:40.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:19:40.207 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:40.206+0000 7f610000f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:40.208 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:40.208+0000 7f610000f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:40.209 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:40.209+0000 7f610000f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:40.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:19:40.535 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:19:40.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:40.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:40.727 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:40.726+0000 7f610000f440 -1 Falling back to public interface 2024-09-15T05:19:40.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:41.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:41.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:41.914 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:19:41.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:19:41.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:41.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:42.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:43.212 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:19:43.211+0000 7f610000f440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:19:43.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:43.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:43.273 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:19:43.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:19:43.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:43.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:43.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:44.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:44.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:44.826 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:19:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:19:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:44.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:46.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:46.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:46.299 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:19:46.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:19:46.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:46.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:46.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:47.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:47.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:47.701 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:19:47.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:19:47.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:47.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:48.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:49.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:49.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:49.260 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:19:49.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:19:49.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:49.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:50.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:50.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:50.635 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:19:50.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:19:50.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:50.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:51.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:52.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:52.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:52.070 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:19:52.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:19:52.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:52.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:52.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:53.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:53.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:53.488 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:19:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:19:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:53.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:54.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:54.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:54.968 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:19:54.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:19:54.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:54.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:55.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:56.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:56.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:56.489 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:19:56.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:19:56.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:56.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:57.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:58.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:58.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:58.116 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:19:58.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:19:58.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:58.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:19:58.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:19:59.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:19:59.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:19:59.827 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:19:59.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:19:59.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:19:59.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:20:00.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:01.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:01.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:01.664 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:20:01.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:20:01.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:01.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:20:02.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:03.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:03.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:03.450 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:20:03.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:20:03.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:03.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:20:04.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:05.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:05.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:05.301 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:20:05.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:20:05.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:05.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:20:05.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:06.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:06.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:06.723 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:20:06.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:20:06.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:06.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:20:07.126 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/373976,v1:127.0.0.1:6803/373976] [v2:127.0.0.1:6804/373976,v1:127.0.0.1:6805/373976] exists,up 52173352-e507-40bc-a6b6-970a77fb31ea 2024-09-15T05:20:07.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:20:07.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:20:07.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:20:07.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:20:07.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:20:07.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:20:07.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:20:07.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:20:07.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:20:07.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:20:07.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:20:07.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:20:07.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:20:07.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:20:07.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6e79056f-aa84-4536-95b9-8ade9de17589 2024-09-15T05:20:07.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6e79056f-aa84-4536-95b9-8ade9de17589' 2024-09-15T05:20:07.140 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 6e79056f-aa84-4536-95b9-8ade9de17589 2024-09-15T05:20:07.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:20:07.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHbuZmFqGUCBAA0sAxnccSiLCIctUlfBrLOA== 2024-09-15T05:20:07.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHbuZmFqGUCBAA0sAxnccSiLCIctUlfBrLOA=="}' 2024-09-15T05:20:07.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6e79056f-aa84-4536-95b9-8ade9de17589 -i td/osd-recovery-stats/1/new.json 2024-09-15T05:20:07.671 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:20:07.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:20:07.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHbuZmFqGUCBAA0sAxnccSiLCIctUlfBrLOA== --osd-uuid 6e79056f-aa84-4536-95b9-8ade9de17589 2024-09-15T05:20:07.706 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.705+0000 7f927fd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:07.708 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.707+0000 7f927fd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:07.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.708+0000 7f927fd21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:07.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.708+0000 7f927fd21440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:20:07.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.708+0000 7f927fd21440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:20:07.709 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.708+0000 7f927fd21440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:20:07.761 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:07.760+0000 7f927fd21440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:20:13.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:20:13.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:20:13.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:20:13.650 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:20:13.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:20:14.133 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:20:14.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:20:14.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:20:14.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:20:14.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:20:14.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:20:14.155 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:14.155+0000 7f7fcc921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:14.157 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:14.157+0000 7f7fcc921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:14.158 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:14.158+0000 7f7fcc921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:14.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:14.481 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:20:14.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:20:14.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:14.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:14.685 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:14.685+0000 7f7fcc921440 -1 Falling back to public interface 2024-09-15T05:20:14.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:15.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:15.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:15.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:20:15.839 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:20:15.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:15.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:16.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:17.196 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:17.195+0000 7f7fcc921440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:20:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:17.249 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:20:17.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:20:17.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:17.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:17.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:18.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:18.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:18.866 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:20:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:20:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:18.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:19.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:20.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:20.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:20.314 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:20:20.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:20:20.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:20.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:20.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:21.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:21.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:21.686 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:20:21.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:20:21.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:21.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:22.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:23.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:23.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:23.130 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:20:23.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:20:23.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:23.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:23.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:24.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:24.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:24.508 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:20:24.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:20:24.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:24.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:24.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:25.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:25.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:25.959 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:20:25.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:20:25.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:25.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:26.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:27.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:27.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:27.457 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:20:27.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:20:27.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:27.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:27.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:28.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:28.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:28.983 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:20:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:20:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:28.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:29.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:30.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:30.340 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:20:30.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:20:30.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:30.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:31.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:31.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:31.979 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:20:31.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:20:31.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:31.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:32.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:33.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:33.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:33.662 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:20:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:20:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:34.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:35.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:35.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:35.381 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:20:35.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:20:35.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:35.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:36.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:37.264 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:20:37.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:20:37.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:37.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:38.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:39.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:39.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:39.317 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:20:39.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:20:39.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:39.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:40.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:41.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:41.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:41.190 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:20:41.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:20:41.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:41.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:20:41.562 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/375730,v1:127.0.0.1:6811/375730] [v2:127.0.0.1:6812/375730,v1:127.0.0.1:6813/375730] exists,up 6e79056f-aa84-4536-95b9-8ade9de17589 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:20:41.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:20:41.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:20:41.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:20:41.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93 2024-09-15T05:20:41.572 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93 2024-09-15T05:20:41.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93' 2024-09-15T05:20:41.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:20:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpbuZmg+mMIhAAQkihonV/vqaJZGoELty0YA== 2024-09-15T05:20:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpbuZmg+mMIhAAQkihonV/vqaJZGoELty0YA=="}' 2024-09-15T05:20:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:20:42.084 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:20:42.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:20:42.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpbuZmg+mMIhAAQkihonV/vqaJZGoELty0YA== --osd-uuid 0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93 2024-09-15T05:20:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.120+0000 7f72c251f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:42.139 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.138+0000 7f72c251f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:42.139 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.139+0000 7f72c251f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:42.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.139+0000 7f72c251f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:20:42.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.139+0000 7f72c251f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:20:42.141 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.139+0000 7f72c251f440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:20:42.195 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:42.194+0000 7f72c251f440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:20:48.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:20:48.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:20:48.082 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:20:48.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:20:48.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:20:48.565 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:20:48.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:20:48.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:20:48.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:20:48.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:20:48.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:20:48.587 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:48.587+0000 7eff8b20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:48.589 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:48.589+0000 7eff8b20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:48.590 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:48.590+0000 7eff8b20f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:20:48.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:48.915 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:20:48.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:20:48.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:48.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:49.109 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:49.109+0000 7eff8b20f440 -1 Falling back to public interface 2024-09-15T05:20:49.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:50.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:50.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:50.264 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:20:50.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:20:50.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:50.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:50.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:51.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:51.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:51.616 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:20:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:20:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:51.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:51.637 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:20:51.636+0000 7eff8b20f440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:20:52.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:53.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:53.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:53.097 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:20:53.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:20:53.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:53.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:53.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:54.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:54.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:54.628 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:20:54.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:20:54.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:54.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:54.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:55.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:56.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:56.000 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:20:56.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:20:56.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:56.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:56.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:57.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:57.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:57.423 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:20:57.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:20:57.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:57.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:57.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:20:58.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:20:58.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:20:58.778 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:20:58.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:20:58.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:20:58.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:20:59.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:00.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:00.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:00.142 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:21:00.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:21:00.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:00.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:00.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:01.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:01.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:01.632 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:21:01.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:21:01.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:01.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:02.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:03.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:03.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:03.048 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:21:03.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:21:03.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:03.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:03.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:04.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:04.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:04.650 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:21:04.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:21:04.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:04.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:05.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:06.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:06.401 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:21:06.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:21:06.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:06.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:07.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:08.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:08.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:08.036 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:21:08.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:21:08.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:08.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:08.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:09.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:09.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:09.739 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:21:09.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:21:09.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:09.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:10.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:11.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:11.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:11.557 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:21:11.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:21:11.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:11.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:12.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:13.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:13.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:13.438 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:21:13.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:21:13.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:13.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:14.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:15.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:15.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:15.308 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:21:15.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:21:15.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:15.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:21:15.690 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/377486,v1:127.0.0.1:6819/377486] [v2:127.0.0.1:6820/377486,v1:127.0.0.1:6821/377486] exists,up 0ce58bd9-ba4d-4520-a43a-f0ee0d03ee93 2024-09-15T05:21:15.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:21:15.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:21:15.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:21:15.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-09-15T05:21:15.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-09-15T05:21:15.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:21:15.696 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:21:15.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:21:15.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-09-15T05:21:15.698 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:21:15.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5051647a-6ac1-4331-9dc1-ccba244f9cb1 2024-09-15T05:21:15.698 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 5051647a-6ac1-4331-9dc1-ccba244f9cb1 2024-09-15T05:21:15.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 5051647a-6ac1-4331-9dc1-ccba244f9cb1' 2024-09-15T05:21:15.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:21:15.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLbuZmos8nKhAAetodVs55yKMkEkvUOFY2Zg== 2024-09-15T05:21:15.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLbuZmos8nKhAAetodVs55yKMkEkvUOFY2Zg=="}' 2024-09-15T05:21:15.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5051647a-6ac1-4331-9dc1-ccba244f9cb1 -i td/osd-recovery-stats/3/new.json 2024-09-15T05:21:16.137 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:21:16.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-09-15T05:21:16.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLbuZmos8nKhAAetodVs55yKMkEkvUOFY2Zg== --osd-uuid 5051647a-6ac1-4331-9dc1-ccba244f9cb1 2024-09-15T05:21:16.173 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.173+0000 7fc62d188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:16.175 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.174+0000 7fc62d188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:16.176 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.175+0000 7fc62d188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:16.176 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.176+0000 7fc62d188440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:21:16.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.176+0000 7fc62d188440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:21:16.177 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.176+0000 7fc62d188440 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-09-15T05:21:16.214 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:16.213+0000 7fc62d188440 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-09-15T05:21:21.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-09-15T05:21:21.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:21:21.844 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:21:21.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:21:21.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:21:22.349 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:21:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:21:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:21:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:21:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:21:22.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:21:22.372 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:22.370+0000 7f5a98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:22.373 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:22.372+0000 7f5a98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:22.374 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:22.373+0000 7f5a98e09440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:21:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:21:22.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:21:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:21:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:21:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:22.702 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:21:22.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:21:22.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:22.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:22.894 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:22.893+0000 7f5a98e09440 -1 Falling back to public interface 2024-09-15T05:21:23.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:24.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:24.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:24.055 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:21:24.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:21:24.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:24.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:24.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:25.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:25.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:25.415 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:21:25.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:21:25.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:25.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:25.434 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:25.432+0000 7f5a98e09440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:21:25.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:26.810 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:21:26.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:21:26.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:26.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:27.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:28.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:28.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:28.259 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:21:28.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:21:28.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:28.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:28.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:29.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:29.704 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:21:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:21:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:29.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:31.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:31.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:31.187 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:21:31.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:21:31.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:31.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:31.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:32.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:32.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:32.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:21:32.585 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:21:32.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:32.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:32.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:33.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:33.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:33.949 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:21:33.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:21:33.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:33.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:34.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:35.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:35.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:35.471 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:21:35.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:21:35.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:35.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:35.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:36.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:36.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:36.830 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:21:36.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:21:36.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:36.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:37.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:38.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:38.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:38.260 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:21:38.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:21:38.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:38.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:38.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:39.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:39.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:39.954 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:21:39.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:21:39.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:39.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:40.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:41.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:41.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:41.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:21:41.644 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:21:41.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:41.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:42.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:43.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:43.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:43.276 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:21:43.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:21:43.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:43.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:43.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:44.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:21:44.858 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:21:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:44.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:45.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:46.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:46.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:46.555 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:21:46.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:21:46.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:46.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:47.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:48.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:48.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:48.366 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:21:48.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:21:48.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:48.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:48.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:49.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:49.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:49.862 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:21:49.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:21:49.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:49.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:21:50.217 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/379240,v1:127.0.0.1:6827/379240] [v2:127.0.0.1:6828/379240,v1:127.0.0.1:6829/379240] exists,up 5051647a-6ac1-4331-9dc1-ccba244f9cb1 2024-09-15T05:21:50.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:21:50.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:21:50.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:21:50.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-09-15T05:21:50.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:21:50.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:21:50.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a82076c1-6477-455f-9e19-9892b0d8e8c7 2024-09-15T05:21:50.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a82076c1-6477-455f-9e19-9892b0d8e8c7' 2024-09-15T05:21:50.225 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 a82076c1-6477-455f-9e19-9892b0d8e8c7 2024-09-15T05:21:50.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:21:50.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDubuZmaAcADhAAEC/9D8lbJqP4omJ7lIlwlQ== 2024-09-15T05:21:50.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDubuZmaAcADhAAEC/9D8lbJqP4omJ7lIlwlQ=="}' 2024-09-15T05:21:50.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a82076c1-6477-455f-9e19-9892b0d8e8c7 -i td/osd-recovery-stats/4/new.json 2024-09-15T05:21:50.746 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:21:50.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-09-15T05:21:50.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDubuZmaAcADhAAEC/9D8lbJqP4omJ7lIlwlQ== --osd-uuid a82076c1-6477-455f-9e19-9892b0d8e8c7 2024-09-15T05:21:50.783 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.782+0000 7f009d329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:50.809 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.809+0000 7f009d329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:50.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.810+0000 7f009d329440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:50.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.810+0000 7f009d329440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:21:50.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.810+0000 7f009d329440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:21:50.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.810+0000 7f009d329440 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-09-15T05:21:50.849 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:50.848+0000 7f009d329440 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-09-15T05:21:56.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-09-15T05:21:56.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:21:56.832 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T05:21:56.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T05:21:56.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:21:57.325 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T05:21:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T05:21:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:21:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:21:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:21:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:21:57.348 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:57.348+0000 7fb943f20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:57.373 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:57.372+0000 7fb943f20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:57.374 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:57.374+0000 7fb943f20440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:57.708 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:21:57.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:21:57.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:57.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:21:57.915 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:21:57.915+0000 7fb943f20440 -1 Falling back to public interface 2024-09-15T05:21:58.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:21:59.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:21:59.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:21:59.072 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:21:59.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:21:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:21:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:21:59.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:00.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:00.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:00.474 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:22:00.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:22:00.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:00.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:00.512 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:00.511+0000 7fb943f20440 -1 osd.4 0 log_to_monitors true 2024-09-15T05:22:00.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:01.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:01.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:01.905 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:22:01.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:22:01.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:01.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:02.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:03.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:03.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:03.430 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:22:03.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:22:03.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:03.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:03.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:04.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:04.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:04.789 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:22:04.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:22:04.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:04.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:05.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:06.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:06.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:06.149 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:22:06.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:22:06.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:06.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:06.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:07.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:07.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:07.613 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:22:07.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:22:07.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:07.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:08.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:09.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:09.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:09.104 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:22:09.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:22:09.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:09.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:09.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:10.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:10.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:10.486 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:22:10.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:22:10.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:10.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:10.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:11.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:11.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:11.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:22:11.961 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:22:11.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:11.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:12.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:13.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:13.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:13.433 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:22:13.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:22:13.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:13.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:15.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:15.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:15.156 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:22:15.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:22:15.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:15.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:16.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:17.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:17.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:17.327 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:22:17.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:22:17.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:17.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:18.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:19.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:19.351 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:22:19.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:19.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:22:19.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:19.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:21.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:21.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:21.253 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:22:21.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:22:21.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:21.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:22.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:23.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:23.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:23.074 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:22:23.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:22:23.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:23.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:23.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:24.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:24.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:24.780 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:22:24.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:22:24.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:24.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:22:25.142 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/381019,v1:127.0.0.1:6835/381019] [v2:127.0.0.1:6836/381019,v1:127.0.0.1:6837/381019] exists,up a82076c1-6477-455f-9e19-9892b0d8e8c7 2024-09-15T05:22:25.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:22:25.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:22:25.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:22:25.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-09-15T05:22:25.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-09-15T05:22:25.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:22:25.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:22:25.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a085b518-6e00-4eb9-a1b2-8ff14f778b01 2024-09-15T05:22:25.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 a085b518-6e00-4eb9-a1b2-8ff14f778b01' 2024-09-15T05:22:25.150 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 a085b518-6e00-4eb9-a1b2-8ff14f778b01 2024-09-15T05:22:25.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:22:25.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARb+Zm2/+CCRAAfnduC+m/L3Gyz5D5BVUKqg== 2024-09-15T05:22:25.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARb+Zm2/+CCRAAfnduC+m/L3Gyz5D5BVUKqg=="}' 2024-09-15T05:22:25.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a085b518-6e00-4eb9-a1b2-8ff14f778b01 -i td/osd-recovery-stats/5/new.json 2024-09-15T05:22:25.779 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:22:25.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-09-15T05:22:25.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARb+Zm2/+CCRAAfnduC+m/L3Gyz5D5BVUKqg== --osd-uuid a085b518-6e00-4eb9-a1b2-8ff14f778b01 2024-09-15T05:22:25.815 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.814+0000 7fd076420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:25.817 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.816+0000 7fd076420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:25.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.817+0000 7fd076420440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:25.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.817+0000 7fd076420440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:22:25.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.818+0000 7fd076420440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:22:25.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.818+0000 7fd076420440 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-09-15T05:22:25.894 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:25.894+0000 7fd076420440 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-09-15T05:22:31.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-09-15T05:22:31.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:22:31.707 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:22:31.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:22:31.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:22:32.190 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:22:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:22:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:22:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:22:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:22:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:22:32.213 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:32.213+0000 7f4cefe16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:32.215 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:32.214+0000 7f4cefe16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:32.216 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:32.215+0000 7f4cefe16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:22:32.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:22:32.541 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:22:32.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:32.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:32.736 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:32.735+0000 7f4cefe16440 -1 Falling back to public interface 2024-09-15T05:22:32.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:33.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:33.890 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:22:33.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:33.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:22:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:34.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:35.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:22:35.202+0000 7f4cefe16440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:22:35.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:35.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:35.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:22:35.246 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:22:35.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:35.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:35.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:36.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:36.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:36.890 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:22:36.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:22:36.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:36.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:37.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:38.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:38.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:38.352 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:22:38.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:22:38.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:38.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:38.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:39.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:39.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:39.719 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:22:39.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:22:39.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:39.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:40.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:41.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:41.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:41.126 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:22:41.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:22:41.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:41.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:42.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:42.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:42.525 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:22:42.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:22:42.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:42.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:43.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:44.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:44.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:44.026 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:22:44.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:22:44.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:44.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:44.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:45.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:45.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:45.445 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:22:45.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:22:45.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:45.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:45.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:46.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:46.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:46.846 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:22:46.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:22:46.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:46.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:47.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:48.479 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:22:48.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:22:48.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:48.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:49.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:50.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:50.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:50.305 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:22:50.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:22:50.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:50.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:51.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:52.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:52.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:52.118 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:22:52.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:22:52.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:52.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:53.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:54.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:54.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:54.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:22:54.226 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:22:54.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:54.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:54.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:55.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:55.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:55.957 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:22:55.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:22:55.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:55.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:56.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:57.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:57.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:57.353 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:22:57.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:22:57.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:57.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:58.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:22:59.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:22:59.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:22:59.275 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:22:59.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:22:59.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:22:59.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:22:59.648 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/382774,v1:127.0.0.1:6843/382774] [v2:127.0.0.1:6844/382774,v1:127.0.0.1:6845/382774] exists,up a085b518-6e00-4eb9-a1b2-8ff14f778b01 2024-09-15T05:22:59.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:22:59.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:22:59.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:22:59.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-09-15T05:22:59.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T05:23:00.209 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:23:00.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:23:01.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-09-15T05:23:02.411 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T05:23:02.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-09-15T05:23:02.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:23:02.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:23:02.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:23:02.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:23:02.423 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:23:02.424 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:23:02.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:23:02.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:23:02.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:23:02.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:23:03.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:23:03.098 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:23:03.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-15T05:23:03.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-15T05:23:03.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T05:23:03.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:23:03.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672999 2024-09-15T05:23:03.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672999 2024-09-15T05:23:03.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672999' 2024-09-15T05:23:03.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:23:03.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509466 2024-09-15T05:23:03.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509466 2024-09-15T05:23:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672999 2-64424509466' 2024-09-15T05:23:03.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:23:03.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345941 2024-09-15T05:23:03.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345941 2024-09-15T05:23:03.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672999 2-64424509466 3-85899345941' 2024-09-15T05:23:03.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:23:03.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182413 2024-09-15T05:23:03.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182413 2024-09-15T05:23:03.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672999 2-64424509466 3-85899345941 4-107374182413' 2024-09-15T05:23:03.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:03.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018889 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018889 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672999 2-64424509466 3-85899345941 4-107374182413 5-128849018889' 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-15T05:23:03.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:03.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:23:03.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-15T05:23:03.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-15T05:23:03.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T05:23:03.733 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836524 2024-09-15T05:23:03.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:23:04.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-15T05:23:04.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:04.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672999 2024-09-15T05:23:04.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:04.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:23:04.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672999 2024-09-15T05:23:04.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:04.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672999 2024-09-15T05:23:04.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-09-15T05:23:04.100 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672999 2024-09-15T05:23:04.100 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:23:04.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672997 -lt 42949672999 2024-09-15T05:23:04.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:23:05.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:23:05.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:23:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672999 -lt 42949672999 2024-09-15T05:23:05.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:05.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509466 2024-09-15T05:23:05.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:05.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:23:05.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509466 2024-09-15T05:23:05.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:05.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509466 2024-09-15T05:23:05.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-09-15T05:23:05.826 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509466 2024-09-15T05:23:05.826 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:23:06.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509466 -lt 64424509466 2024-09-15T05:23:06.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:06.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345941 2024-09-15T05:23:06.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:06.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:23:06.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345941 2024-09-15T05:23:06.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:06.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345941 2024-09-15T05:23:06.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-09-15T05:23:06.190 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345941 2024-09-15T05:23:06.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:23:06.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345941 2024-09-15T05:23:06.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:06.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182413 2024-09-15T05:23:06.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:06.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:23:06.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182413 2024-09-15T05:23:06.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:06.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182413 2024-09-15T05:23:06.547 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182413 2024-09-15T05:23:06.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182413' 2024-09-15T05:23:06.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:23:06.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182414 -lt 107374182413 2024-09-15T05:23:06.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:06.931 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018889 2024-09-15T05:23:06.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:06.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:23:06.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018889 2024-09-15T05:23:06.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:06.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018889 2024-09-15T05:23:06.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2024-09-15T05:23:06.934 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018889 2024-09-15T05:23:06.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:23:07.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018889 -lt 128849018889 2024-09-15T05:23:07.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:23:07.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:07.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:07.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:08.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:23:08.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:08.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:08.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:23:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:23:08.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:23:08.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-09-15T05:23:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:08.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-09-15T05:23:08.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:08.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-09-15T05:23:08.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:08.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-09-15T05:23:08.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:08.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-09-15T05:23:09.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-09-15T05:23:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-09-15T05:23:09.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-09-15T05:23:09.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-09-15T05:23:09.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-09-15T05:23:09.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-09-15T05:23:09.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-09-15T05:23:09.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-09-15T05:23:09.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-09-15T05:23:09.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:09.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-09-15T05:23:10.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-09-15T05:23:10.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-09-15T05:23:10.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-09-15T05:23:10.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-09-15T05:23:10.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-09-15T05:23:10.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-09-15T05:23:10.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-09-15T05:23:10.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-09-15T05:23:10.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-09-15T05:23:10.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-09-15T05:23:10.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:10.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-09-15T05:23:11.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-09-15T05:23:11.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-09-15T05:23:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-09-15T05:23:11.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-09-15T05:23:11.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-09-15T05:23:11.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-09-15T05:23:11.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:11.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-09-15T05:23:12.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-09-15T05:23:12.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-09-15T05:23:12.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-09-15T05:23:12.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-09-15T05:23:12.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-09-15T05:23:12.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-09-15T05:23:12.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-09-15T05:23:12.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-09-15T05:23:12.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-09-15T05:23:12.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-09-15T05:23:12.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:12.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-09-15T05:23:13.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-09-15T05:23:13.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-09-15T05:23:13.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-09-15T05:23:13.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-09-15T05:23:13.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-09-15T05:23:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-09-15T05:23:13.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-09-15T05:23:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-09-15T05:23:13.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-09-15T05:23:13.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:13.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-09-15T05:23:14.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-09-15T05:23:14.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-09-15T05:23:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-09-15T05:23:14.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-09-15T05:23:14.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-09-15T05:23:14.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-09-15T05:23:14.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-09-15T05:23:14.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-09-15T05:23:14.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-09-15T05:23:14.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:14.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-09-15T05:23:15.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-09-15T05:23:15.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-09-15T05:23:15.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-09-15T05:23:15.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-09-15T05:23:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-09-15T05:23:15.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-09-15T05:23:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-09-15T05:23:15.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-09-15T05:23:15.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-09-15T05:23:15.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:15.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-09-15T05:23:16.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-09-15T05:23:16.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-09-15T05:23:16.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-09-15T05:23:16.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-09-15T05:23:16.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-09-15T05:23:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-09-15T05:23:16.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-09-15T05:23:16.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-09-15T05:23:16.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-09-15T05:23:16.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:16.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-09-15T05:23:17.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-09-15T05:23:17.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-09-15T05:23:17.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-09-15T05:23:17.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-09-15T05:23:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-09-15T05:23:17.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-09-15T05:23:17.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-09-15T05:23:17.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-09-15T05:23:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:17.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-09-15T05:23:18.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-09-15T05:23:18.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-09-15T05:23:18.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-09-15T05:23:18.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-09-15T05:23:18.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-09-15T05:23:18.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-09-15T05:23:18.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-09-15T05:23:18.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-09-15T05:23:18.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-09-15T05:23:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-09-15T05:23:19.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-09-15T05:23:19.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-09-15T05:23:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-09-15T05:23:19.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-09-15T05:23:19.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-09-15T05:23:19.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-09-15T05:23:19.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-09-15T05:23:19.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-09-15T05:23:19.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-09-15T05:23:19.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:19.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-09-15T05:23:20.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-09-15T05:23:20.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-09-15T05:23:20.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-09-15T05:23:20.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-09-15T05:23:20.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-09-15T05:23:20.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-09-15T05:23:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-09-15T05:23:20.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-09-15T05:23:20.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-09-15T05:23:20.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:20.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-09-15T05:23:21.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-09-15T05:23:21.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-09-15T05:23:21.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-09-15T05:23:21.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-09-15T05:23:21.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-09-15T05:23:21.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-09-15T05:23:21.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-09-15T05:23:21.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-09-15T05:23:21.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-09-15T05:23:21.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:21.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-09-15T05:23:22.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-09-15T05:23:22.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-09-15T05:23:22.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-09-15T05:23:22.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-09-15T05:23:22.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-09-15T05:23:22.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-09-15T05:23:22.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-09-15T05:23:22.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-09-15T05:23:22.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-09-15T05:23:22.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:22.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-09-15T05:23:23.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-09-15T05:23:23.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-09-15T05:23:23.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-09-15T05:23:23.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-09-15T05:23:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-09-15T05:23:23.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-09-15T05:23:23.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-09-15T05:23:23.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-09-15T05:23:23.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-09-15T05:23:23.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-09-15T05:23:23.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:23.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-09-15T05:23:24.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-09-15T05:23:24.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-09-15T05:23:24.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-09-15T05:23:24.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-09-15T05:23:24.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-09-15T05:23:24.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-09-15T05:23:24.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-09-15T05:23:24.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-09-15T05:23:24.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-09-15T05:23:24.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:24.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-09-15T05:23:25.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-09-15T05:23:25.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-09-15T05:23:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-09-15T05:23:25.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-09-15T05:23:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-09-15T05:23:25.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-09-15T05:23:25.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-09-15T05:23:25.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-09-15T05:23:25.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-09-15T05:23:25.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:25.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-09-15T05:23:26.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-09-15T05:23:26.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-09-15T05:23:26.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-09-15T05:23:26.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-09-15T05:23:26.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-09-15T05:23:26.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-09-15T05:23:26.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-09-15T05:23:26.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-09-15T05:23:26.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-09-15T05:23:26.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:26.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-09-15T05:23:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-09-15T05:23:27.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-09-15T05:23:27.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-09-15T05:23:27.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-09-15T05:23:27.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-09-15T05:23:27.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-09-15T05:23:27.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-09-15T05:23:27.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-09-15T05:23:27.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-09-15T05:23:27.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:27.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-09-15T05:23:28.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-09-15T05:23:28.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-09-15T05:23:28.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-09-15T05:23:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-09-15T05:23:28.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-09-15T05:23:28.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-09-15T05:23:28.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-09-15T05:23:28.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-09-15T05:23:28.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-09-15T05:23:28.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:23:28.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:23:28.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:23:28.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:23:29.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-09-15T05:23:29.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-09-15T05:23:29.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:23:29.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:23:29.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:23:29.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:23:29.403 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:23:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:23:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:23:29.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-15T05:23:29.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-09-15T05:23:29.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-15T05:23:30.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-09-15T05:23:30.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-09-15T05:23:30.843 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:23:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-09-15T05:23:31.478 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already out. osd.0 is already out. 2024-09-15T05:23:31.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-09-15T05:23:32.824 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 4 2024-09-15T05:23:32.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-09-15T05:23:33.924 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:23:33.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-09-15T05:23:33.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:23:33.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:23:33.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:23:33.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:23:34.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-09-15T05:23:34.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-09-15T05:23:34.433 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:23:34.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-09-15T05:23:36.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-09-15T05:23:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:23:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:23:36.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:23:36.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:23:36.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:23:36.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:23:36.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:23:36.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:23:36.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:23:36.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:23:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:23:36.897 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:36.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:23:37.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836532 2024-09-15T05:23:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836532 2024-09-15T05:23:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532' 2024-09-15T05:23:37.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:37.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:23:37.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673008 2024-09-15T05:23:37.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673008 2024-09-15T05:23:37.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673008' 2024-09-15T05:23:37.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:37.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:23:37.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509475 2024-09-15T05:23:37.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509475 2024-09-15T05:23:37.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673008 2-64424509475' 2024-09-15T05:23:37.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:37.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:23:37.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345950 2024-09-15T05:23:37.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345950 2024-09-15T05:23:37.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673008 2-64424509475 3-85899345950' 2024-09-15T05:23:37.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:37.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:23:37.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182422 2024-09-15T05:23:37.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182422 2024-09-15T05:23:37.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673008 2-64424509475 3-85899345950 4-107374182422' 2024-09-15T05:23:37.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:23:37.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:23:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018897 2024-09-15T05:23:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018897 2024-09-15T05:23:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836532 1-42949673008 2-64424509475 3-85899345950 4-107374182422 5-128849018897' 2024-09-15T05:23:37.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:37.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836532 2024-09-15T05:23:37.736 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:37.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:23:37.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836532 2024-09-15T05:23:37.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:37.738 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836532 2024-09-15T05:23:37.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836532 2024-09-15T05:23:37.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836532' 2024-09-15T05:23:37.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:23:38.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836532 -lt 21474836532 2024-09-15T05:23:38.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:38.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673008 2024-09-15T05:23:38.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:38.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:23:38.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673008 2024-09-15T05:23:38.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:38.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673008 2024-09-15T05:23:38.123 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673008 2024-09-15T05:23:38.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-15T05:23:38.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:23:38.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-15T05:23:38.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:38.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509475 2024-09-15T05:23:38.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:38.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:23:38.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509475 2024-09-15T05:23:38.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:38.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509475 2024-09-15T05:23:38.616 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509475 2024-09-15T05:23:38.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509475' 2024-09-15T05:23:38.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:23:39.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509473 -lt 64424509475 2024-09-15T05:23:39.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:23:40.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:23:40.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:23:40.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509475 -lt 64424509475 2024-09-15T05:23:40.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:40.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345950 2024-09-15T05:23:40.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:40.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:23:40.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345950 2024-09-15T05:23:40.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:40.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345950 2024-09-15T05:23:40.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345950' 2024-09-15T05:23:40.423 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345950 2024-09-15T05:23:40.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:23:40.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345950 -lt 85899345950 2024-09-15T05:23:40.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:40.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182422 2024-09-15T05:23:40.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:40.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:23:40.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182422 2024-09-15T05:23:40.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:40.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182422 2024-09-15T05:23:40.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182422' 2024-09-15T05:23:40.950 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182422 2024-09-15T05:23:40.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:23:41.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182422 -lt 107374182422 2024-09-15T05:23:41.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:23:41.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018897 2024-09-15T05:23:41.332 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:23:41.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:23:41.333 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018897 2024-09-15T05:23:41.333 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:23:41.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018897 2024-09-15T05:23:41.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-09-15T05:23:41.335 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018897 2024-09-15T05:23:41.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:23:41.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018898 -lt 128849018897 2024-09-15T05:23:41.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:23:41.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:41.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:42.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:42.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:42.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:42.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:43.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:43.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:23:43.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:23:43.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:23:43.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:43.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:43.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:44.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:44.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:44.439 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:45.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-15T05:23:45.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-15T05:23:45.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:45.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:45.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:45.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:45.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:45.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:45.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:45.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:45.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:45.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:46.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:46.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:46.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-09-15T05:23:46.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-09-15T05:23:46.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:46.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:46.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:46.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:46.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:46.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:47.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:47.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:47.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:47.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:47.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:48.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:23:48.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:23:48.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:48.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:48.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:48.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:48.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:48.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:48.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:48.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:48.660 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:49.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:49.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:49.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:49.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:49.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:23:49.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:23:49.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:49.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:49.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:49.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:50.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:50.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:50.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:50.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:50.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:50.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:50.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:50.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:50.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:50.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:50.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:51.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:51.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:51.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:51.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:23:51.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:23:51.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:51.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:51.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:51.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:51.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:51.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:52.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:52.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:52.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:52.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:52.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:53.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-09-15T05:23:53.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-09-15T05:23:53.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:53.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:53.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:53.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:53.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:53.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:53.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:53.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:53.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:53.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:54.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:54.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:54.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:54.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:23:54.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:23:54.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:54.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:54.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:54.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:54.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:55.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:55.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:55.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:55.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:55.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:56.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:23:56.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:23:56.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:56.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:56.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:56.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:56.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:56.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:56.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:57.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:57.470 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:23:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:23:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:58.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:58.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:58.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:58.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:58.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:58.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:58.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:58.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:23:59.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:23:59.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:23:59.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:23:59.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:23:59.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:23:59.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:23:59.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:23:59.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:23:59.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:23:59.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:23:59.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:23:59.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:23:59.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:23:59.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:00.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:00.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:01.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:24:01.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:24:01.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:01.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:01.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:01.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:01.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:01.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:01.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:01.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:01.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:01.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:01.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:01.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:01.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:01.444 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:01.959 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:01.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:02.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:24:02.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:24:02.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:02.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:02.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:02.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:02.648 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:03.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:03.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:03.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:03.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:03.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:03.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:03.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:03.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:04.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:24:04.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:24:04.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:04.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:04.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:04.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:04.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:04.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:04.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:04.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:04.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:04.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:04.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:04.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:04.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:04.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:04.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:05.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:05.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:05.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:24:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:24:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:05.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:05.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:05.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:06.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:06.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:06.181 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:06.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:06.710 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:06.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:07.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-09-15T05:24:07.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-09-15T05:24:07.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:07.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:07.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:07.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:07.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:07.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:07.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:07.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:08.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:08.339 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:08.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-09-15T05:24:08.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-09-15T05:24:08.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:08.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:08.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:09.027 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:09.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:09.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:09.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:09.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:09.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:09.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:10.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:10.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:10.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:24:10.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:24:10.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:10.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:10.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:10.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:11.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:24:11.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:11.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:11.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:24:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:24:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:24:11.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:24:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:24:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:24:11.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:24:11.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:24:11.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:24:12.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-09-15T05:24:12.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-09-15T05:24:12.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:24:12.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:24:12.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:24:12.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:24:12.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:24:12.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:24:12.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:24:12.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:24:12.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:24:13.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:24:13.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:24:13.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:24:13.109 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-15T05:24:13.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-09-15T05:24:13.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-09-15T05:24:13.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:24:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:24:13.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-09-15T05:24:13.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:24:13.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.139 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.139 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-15T05:24:13.139 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:24:13.140 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:24:13.140 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:24:13.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:24:13.144 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:24:13.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.155 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.155 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-15T05:24:13.155 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:24:13.155 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:24:13.155 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:24:13.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.159 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.159 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:24:13.159 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:24:13.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-09-15T05:24:13.172 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-09-15T05:24:13.173 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-09-15T05:24:13.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-09-15T05:24:13.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-09-15T05:24:13.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:24:13.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-09-15T05:24:13.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:24:13.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:24:13.176 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-09-15T05:24:13.176 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-09-15T05:24:13.177 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:24:13.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:24:13.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-09-15T05:24:13.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:24:13.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:24:13.803 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:24:13.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-09-15T05:24:13.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:24:13.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:24:13.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:24:13.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:24:13.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:24:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:24:19.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:24:19.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:24:19.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:24:19.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:24:19.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:24:19.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:24:19.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:24:19.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:24:19.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:24:19.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:24:19.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:24:19.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:24:19.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:24:19.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:24:19.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:24:19.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:24:19.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:24:19.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:24:19.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:24:19.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:24:19.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-09-15T05:24:19.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:24:19.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:24:19.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:24:19.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:24:19.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:24:19.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:24:19.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:24:19.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:24:19.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:24:19.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:24:19.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:24:19.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:24:19.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:24:19.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:24:19.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:24:19.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:24:19.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:24:19.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:24:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:24:19.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:24:19.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-09-15T05:24:19.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:24:19.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.290662 2024-09-15T05:24:19.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:24:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:24:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:24:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-09-15T05:24:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-09-15T05:24:19.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-09-15T05:24:19.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-09-15T05:24:19.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:24:19.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:24:19.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:24:19.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-09-15T05:24:19.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:19.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:24:19.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:24:19.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:24:19.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:24:19.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:24:19.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:24:19.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:24:19.824 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:24:19.824 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:24:19.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get fsid 2024-09-15T05:24:19.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:24:19.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:24:19.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.290662/ceph-mon.a.asok 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:24:19.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.290662/ceph-mon.a.asok config get mon_host 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-09-15T05:24:19.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:20.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:20.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:24:20.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:24:20.841 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-09-15T05:24:20.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-09-15T05:24:20.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:24:20.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-09-15T05:24:20.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:24:20.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:24:20.847 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:24:20.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc4e0972-9a8c-4561-ab2e-0e0efd3b537c 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 cc4e0972-9a8c-4561-ab2e-0e0efd3b537c' 2024-09-15T05:24:20.849 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 cc4e0972-9a8c-4561-ab2e-0e0efd3b537c 2024-09-15T05:24:20.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:24:20.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCEb+Zm1RhCMxAAOA3d9fyquVQXfkhV6KqMNA== 2024-09-15T05:24:20.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCEb+Zm1RhCMxAAOA3d9fyquVQXfkhV6KqMNA=="}' 2024-09-15T05:24:20.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc4e0972-9a8c-4561-ab2e-0e0efd3b537c -i td/osd-recovery-stats/0/new.json 2024-09-15T05:24:21.160 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:24:21.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-09-15T05:24:21.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCEb+Zm1RhCMxAAOA3d9fyquVQXfkhV6KqMNA== --osd-uuid cc4e0972-9a8c-4561-ab2e-0e0efd3b537c 2024-09-15T05:24:21.195 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.195+0000 7f115be4d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:21.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.205+0000 7f115be4d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:21.207 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.206+0000 7f115be4d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:21.207 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.207+0000 7f115be4d440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:24:21.207 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.207+0000 7f115be4d440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:24:21.208 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.207+0000 7f115be4d440 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-09-15T05:24:21.238 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:21.237+0000 7f115be4d440 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-09-15T05:24:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-09-15T05:24:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:24:26.291 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:24:26.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:24:26.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:24:26.778 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:24:26.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:24:26.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:24:26.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:24:26.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:24:26.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:24:26.801 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:26.800+0000 7f6fec008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:26.803 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:26.802+0000 7f6fec008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:26.804 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:26.803+0000 7f6fec008440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:24:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:24:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:24:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:24:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:24:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:27.131 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:24:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:24:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:27.326 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:27.324+0000 7f6fec008440 -1 Falling back to public interface 2024-09-15T05:24:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:28.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:28.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:28.525 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:24:28.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:24:28.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:28.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:28.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:29.755 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:29.753+0000 7f6fec008440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:24:29.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:29.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:29.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:24:29.885 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:24:29.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:29.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:30.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:31.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:31.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:31.332 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:24:31.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:24:31.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:31.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:31.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:32.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:32.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:32.737 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:24:32.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:24:32.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:32.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:33.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:34.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:34.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:34.133 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:24:34.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:24:34.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:34.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:34.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:35.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:35.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:35.600 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:24:35.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:24:35.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:35.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:35.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:36.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:36.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:36.957 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:24:36.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:24:36.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:36.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:37.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:38.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:38.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:38.460 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:24:38.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:24:38.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:38.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:38.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:39.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:39.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:39.833 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:24:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:24:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:39.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:40.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:41.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:41.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:41.192 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:24:41.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:24:41.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:41.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:42.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:43.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:43.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:43.176 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:24:43.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:24:43.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:43.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:43.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:44.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:44.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:44.840 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:24:44.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:24:44.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:44.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:45.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:46.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:46.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:46.519 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:24:46.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:24:46.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:46.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:47.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:48.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:48.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:48.470 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:24:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:24:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:48.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:49.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:50.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:50.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:50.418 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:24:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:24:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:50.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:51.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:24:52.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:24:52.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:24:52.246 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:24:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:24:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:24:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:24:52.683 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/391624,v1:127.0.0.1:6803/391624] [v2:127.0.0.1:6804/391624,v1:127.0.0.1:6805/391624] exists,up cc4e0972-9a8c-4561-ab2e-0e0efd3b537c 2024-09-15T05:24:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:24:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:24:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:24:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:24:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-09-15T05:24:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:24:52.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1dc5c264-f341-4c15-815d-bf6247b4f83c 2024-09-15T05:24:52.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1dc5c264-f341-4c15-815d-bf6247b4f83c' 2024-09-15T05:24:52.691 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 1dc5c264-f341-4c15-815d-bf6247b4f83c 2024-09-15T05:24:52.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:24:52.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCkb+ZmG27DKRAAuMuiKeKiy4xR3bkBsP/yUw== 2024-09-15T05:24:52.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCkb+ZmG27DKRAAuMuiKeKiy4xR3bkBsP/yUw=="}' 2024-09-15T05:24:52.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1dc5c264-f341-4c15-815d-bf6247b4f83c -i td/osd-recovery-stats/1/new.json 2024-09-15T05:24:53.208 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:24:53.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-09-15T05:24:53.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCkb+ZmG27DKRAAuMuiKeKiy4xR3bkBsP/yUw== --osd-uuid 1dc5c264-f341-4c15-815d-bf6247b4f83c 2024-09-15T05:24:53.244 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.243+0000 7f5048f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:53.245 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.245+0000 7f5048f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:53.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.246+0000 7f5048f23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:24:53.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.246+0000 7f5048f23440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:24:53.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.246+0000 7f5048f23440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:24:53.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.246+0000 7f5048f23440 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-09-15T05:24:53.323 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:24:53.322+0000 7f5048f23440 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-09-15T05:24:59.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-09-15T05:24:59.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:24:59.772 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:24:59.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:24:59.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:25:00.252 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:25:00.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:25:00.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:25:00.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:25:00.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:25:00.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:25:00.275 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:00.274+0000 7fc6c988d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:00.277 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:00.276+0000 7fc6c988d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:00.280 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:00.277+0000 7fc6c988d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:00.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:25:00.636 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:25:00.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:00.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:00.796 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:00.795+0000 7fc6c988d440 -1 Falling back to public interface 2024-09-15T05:25:00.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:01.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:01.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:01.995 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:25:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:25:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:02.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:03.385 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:03.384+0000 7fc6c988d440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:25:03.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:03.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:25:03.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:25:03.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:03.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:03.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:04.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:04.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:04.811 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:25:04.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:25:04.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:04.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:05.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:06.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:06.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:06.297 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:25:06.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:25:06.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:06.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:06.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:07.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:07.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:07.648 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:25:07.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:25:07.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:07.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:08.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:09.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:09.129 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:25:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:25:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:09.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:09.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:10.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:10.525 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:25:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:25:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:10.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:10.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:11.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:11.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:11.889 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:25:11.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:25:11.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:11.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:12.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:13.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:13.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:13.420 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:25:13.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:25:13.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:13.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:13.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:14.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:14.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:14.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:25:14.819 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:25:14.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:14.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:15.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:16.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:16.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:16.298 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:25:16.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:25:16.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:16.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:17.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:17.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:17.795 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:25:17.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:25:17.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:17.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:18.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:19.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:19.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:19.593 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:25:19.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:25:19.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:19.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:20.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:21.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:21.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:21.446 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:25:21.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:25:21.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:21.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:22.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:23.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:23.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:23.283 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:25:23.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:25:23.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:23.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:24.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:25.119 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:25:25.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:25:25.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:25.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:26.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:27.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:27.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:27.130 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:25:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:25:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:27.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:25:27.518 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/393353,v1:127.0.0.1:6811/393353] [v2:127.0.0.1:6812/393353,v1:127.0.0.1:6813/393353] exists,up 1dc5c264-f341-4c15-815d-bf6247b4f83c 2024-09-15T05:25:27.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:25:27.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:25:27.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:25:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:25:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9287835-fa19-44c4-82a6-6ad0ba57c8c0 2024-09-15T05:25:27.526 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 b9287835-fa19-44c4-82a6-6ad0ba57c8c0 2024-09-15T05:25:27.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b9287835-fa19-44c4-82a6-6ad0ba57c8c0' 2024-09-15T05:25:27.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:25:27.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHb+ZmoebtHxAALnjwPJRa4rCNRsg+xBgYZQ== 2024-09-15T05:25:27.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHb+ZmoebtHxAALnjwPJRa4rCNRsg+xBgYZQ=="}' 2024-09-15T05:25:27.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9287835-fa19-44c4-82a6-6ad0ba57c8c0 -i td/osd-recovery-stats/2/new.json 2024-09-15T05:25:28.012 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:25:28.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-09-15T05:25:28.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHb+ZmoebtHxAALnjwPJRa4rCNRsg+xBgYZQ== --osd-uuid b9287835-fa19-44c4-82a6-6ad0ba57c8c0 2024-09-15T05:25:28.049 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.048+0000 7f708174c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:28.050 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.049+0000 7f708174c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:28.051 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.050+0000 7f708174c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:28.052 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.051+0000 7f708174c440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:25:28.052 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.051+0000 7f708174c440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:25:28.052 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.051+0000 7f708174c440 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-09-15T05:25:28.086 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:28.085+0000 7f708174c440 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-09-15T05:25:34.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-09-15T05:25:34.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:25:34.264 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:25:34.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:25:34.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:25:34.744 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:25:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:25:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=592e9c90-7756-4550-bee1-5d7a63895947 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.290662/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:25:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:25:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:25:34.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:25:34.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:34.766+0000 7f5fe2d28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:34.769 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:34.768+0000 7f5fe2d28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:34.770 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:34.769+0000 7f5fe2d28440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:25:35.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:35.098 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:25:35.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:25:35.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:35.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:35.282 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:35.281+0000 7f5fe2d28440 -1 Falling back to public interface 2024-09-15T05:25:35.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:36.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:36.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:36.471 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:25:36.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:25:36.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:36.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:36.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:37.770 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:25:37.770+0000 7f5fe2d28440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:25:37.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:37.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:37.866 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:25:37.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:25:37.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:37.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:38.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:39.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:39.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:39.491 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:25:39.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:25:39.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:39.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:39.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:40.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:40.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:40.888 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:25:40.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:25:40.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:40.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:42.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:42.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:42.388 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:25:42.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:25:42.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:42.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:42.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:43.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:43.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:43.959 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:25:43.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:25:43.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:43.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:44.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:45.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:45.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:45.352 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:25:45.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:25:45.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:45.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:45.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:46.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:46.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:46.784 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:25:46.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:25:46.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:46.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:47.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:48.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:48.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:48.325 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:25:48.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:25:48.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:48.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:48.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:49.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:49.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:49.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:25:49.811 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:25:49.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:49.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:50.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:51.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:51.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:51.197 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:25:51.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:25:51.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:51.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:52.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:53.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:53.265 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:25:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:25:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:53.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:53.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:54.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:54.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:54.982 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:25:54.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:25:54.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:54.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:55.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:56.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:56.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:56.778 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:25:56.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:25:56.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:56.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:57.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:25:58.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:25:58.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:25:58.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:25:58.535 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:25:58.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:25:58.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:25:59.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:00.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:00.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:00.405 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:26:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:26:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:00.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:26:01.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:02.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:02.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:02.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:26:02.244 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:26:02.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:02.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:26:02.597 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/395108,v1:127.0.0.1:6819/395108] [v2:127.0.0.1:6820/395108,v1:127.0.0.1:6821/395108] exists,up b9287835-fa19-44c4-82a6-6ad0ba57c8c0 2024-09-15T05:26:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:26:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:26:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:26:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-09-15T05:26:02.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T05:26:03.268 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:26:03.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:26:04.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-09-15T05:26:05.287 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 1 2024-09-15T05:26:05.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-09-15T05:26:05.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:26:05.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:26:05.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:26:05.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:26:05.300 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:26:05.300 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:26:05.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:26:05.300 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:26:05.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:26:05.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:05.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:26:05.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836500 2024-09-15T05:26:05.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836500 2024-09-15T05:26:05.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500' 2024-09-15T05:26:05.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:05.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:26:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-15T05:26:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-15T05:26:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672972' 2024-09-15T05:26:06.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:06.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:26:06.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509448 2024-09-15T05:26:06.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509448 2024-09-15T05:26:06.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836500 1-42949672972 2-64424509448' 2024-09-15T05:26:06.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:06.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836500 2024-09-15T05:26:06.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:06.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:26:06.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836500 2024-09-15T05:26:06.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:06.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836500 2024-09-15T05:26:06.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-15T05:26:06.197 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836500 2024-09-15T05:26:06.197 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:26:06.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836500 -lt 21474836500 2024-09-15T05:26:06.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:06.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-15T05:26:06.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:06.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:26:06.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-15T05:26:06.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:06.582 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672972 2024-09-15T05:26:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-15T05:26:06.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-15T05:26:06.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:26:06.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-15T05:26:06.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:06.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509448 2024-09-15T05:26:06.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:06.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:26:06.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509448 2024-09-15T05:26:06.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:06.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509448 2024-09-15T05:26:06.938 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509448 2024-09-15T05:26:06.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-09-15T05:26:06.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:26:07.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509448 -lt 64424509448 2024-09-15T05:26:07.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:26:07.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:26:07.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:26:07.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:26:07.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:26:07.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:26:08.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:26:08.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:26:08.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:26:08.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:26:08.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:26:08.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:26:08.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:26:08.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-09-15T05:26:08.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-09-15T05:26:08.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-09-15T05:26:08.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-09-15T05:26:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-09-15T05:26:08.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-09-15T05:26:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:08.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-09-15T05:26:09.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-09-15T05:26:09.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-09-15T05:26:09.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-09-15T05:26:09.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-09-15T05:26:09.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-09-15T05:26:09.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-09-15T05:26:09.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-09-15T05:26:09.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-09-15T05:26:09.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-09-15T05:26:09.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-09-15T05:26:09.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-09-15T05:26:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-09-15T05:26:09.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-09-15T05:26:09.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-09-15T05:26:09.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:09.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-09-15T05:26:10.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-09-15T05:26:10.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-09-15T05:26:10.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-09-15T05:26:10.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-09-15T05:26:10.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-09-15T05:26:10.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-09-15T05:26:10.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-09-15T05:26:10.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-09-15T05:26:10.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-09-15T05:26:10.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-09-15T05:26:10.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-09-15T05:26:10.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-09-15T05:26:10.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-09-15T05:26:10.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-09-15T05:26:10.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:10.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-09-15T05:26:11.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-09-15T05:26:11.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-09-15T05:26:11.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-09-15T05:26:11.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-09-15T05:26:11.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-09-15T05:26:11.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-09-15T05:26:11.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-09-15T05:26:11.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-09-15T05:26:11.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-09-15T05:26:11.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-09-15T05:26:11.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-09-15T05:26:11.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-09-15T05:26:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-09-15T05:26:11.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-09-15T05:26:11.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-09-15T05:26:11.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:11.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-09-15T05:26:12.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-09-15T05:26:12.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-09-15T05:26:12.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-09-15T05:26:12.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-09-15T05:26:12.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-09-15T05:26:12.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-09-15T05:26:12.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-09-15T05:26:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-09-15T05:26:12.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-09-15T05:26:12.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-09-15T05:26:12.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-09-15T05:26:12.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-09-15T05:26:12.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-09-15T05:26:12.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-09-15T05:26:12.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:12.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-09-15T05:26:13.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-09-15T05:26:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-09-15T05:26:13.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-09-15T05:26:13.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-09-15T05:26:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-09-15T05:26:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-09-15T05:26:13.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-09-15T05:26:13.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-09-15T05:26:13.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-09-15T05:26:13.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-09-15T05:26:13.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-09-15T05:26:13.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-09-15T05:26:13.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-09-15T05:26:13.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-09-15T05:26:13.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:13.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-09-15T05:26:14.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-09-15T05:26:14.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-09-15T05:26:14.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-09-15T05:26:14.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-09-15T05:26:14.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-09-15T05:26:14.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-09-15T05:26:14.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-09-15T05:26:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-09-15T05:26:14.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-09-15T05:26:14.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-09-15T05:26:14.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-09-15T05:26:14.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-09-15T05:26:14.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-09-15T05:26:14.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-09-15T05:26:14.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:14.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-09-15T05:26:15.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-09-15T05:26:15.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-09-15T05:26:15.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-09-15T05:26:15.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-09-15T05:26:15.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-09-15T05:26:15.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-09-15T05:26:15.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-09-15T05:26:15.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-09-15T05:26:15.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-09-15T05:26:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-09-15T05:26:15.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-09-15T05:26:15.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-09-15T05:26:15.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-09-15T05:26:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-09-15T05:26:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-09-15T05:26:15.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:15.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-09-15T05:26:16.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-09-15T05:26:16.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-09-15T05:26:16.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-09-15T05:26:16.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-09-15T05:26:16.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-09-15T05:26:16.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-09-15T05:26:16.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-09-15T05:26:16.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-09-15T05:26:16.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-09-15T05:26:16.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-09-15T05:26:16.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-09-15T05:26:16.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-09-15T05:26:16.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-09-15T05:26:16.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-09-15T05:26:17.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-09-15T05:26:17.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-09-15T05:26:17.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-09-15T05:26:17.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-09-15T05:26:17.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-09-15T05:26:17.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-09-15T05:26:17.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-09-15T05:26:17.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-09-15T05:26:17.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-09-15T05:26:17.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-09-15T05:26:17.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-09-15T05:26:17.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-09-15T05:26:17.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-09-15T05:26:17.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-09-15T05:26:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-09-15T05:26:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-09-15T05:26:18.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-09-15T05:26:18.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-09-15T05:26:18.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-09-15T05:26:18.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-09-15T05:26:18.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-09-15T05:26:18.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-09-15T05:26:18.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-09-15T05:26:18.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-09-15T05:26:18.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-09-15T05:26:18.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-09-15T05:26:18.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-09-15T05:26:18.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-09-15T05:26:18.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-09-15T05:26:18.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-09-15T05:26:18.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:18.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-09-15T05:26:19.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-09-15T05:26:19.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-09-15T05:26:19.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-09-15T05:26:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-09-15T05:26:19.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-09-15T05:26:19.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-09-15T05:26:19.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-09-15T05:26:19.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-09-15T05:26:19.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-09-15T05:26:19.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-09-15T05:26:19.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-09-15T05:26:19.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-09-15T05:26:19.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-09-15T05:26:19.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-09-15T05:26:19.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:19.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-09-15T05:26:20.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-09-15T05:26:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-09-15T05:26:20.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-09-15T05:26:20.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-09-15T05:26:20.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-09-15T05:26:20.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-09-15T05:26:20.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-09-15T05:26:20.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-09-15T05:26:20.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-09-15T05:26:20.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-09-15T05:26:20.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-09-15T05:26:20.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-09-15T05:26:20.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-09-15T05:26:20.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:20.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-09-15T05:26:21.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-09-15T05:26:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-09-15T05:26:21.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-09-15T05:26:21.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-09-15T05:26:21.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-09-15T05:26:21.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-09-15T05:26:21.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-09-15T05:26:21.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-09-15T05:26:21.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-09-15T05:26:21.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-09-15T05:26:21.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-09-15T05:26:21.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-09-15T05:26:21.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-09-15T05:26:21.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-09-15T05:26:22.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-09-15T05:26:22.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:26:22.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:26:22.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:26:22.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:26:22.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-09-15T05:26:22.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-09-15T05:26:22.372 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-09-15T05:26:22.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-09-15T05:26:22.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-09-15T05:26:22.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:26:22.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-09-15T05:26:22.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-09-15T05:26:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is set 2024-09-15T05:26:23.571 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-09-15T05:26:23.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-09-15T05:26:23.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-09-15T05:26:23.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-09-15T05:26:23.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-09-15T05:26:24.073 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:26:24.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-09-15T05:26:24.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-09-15T05:26:24.723 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 4 2024-09-15T05:26:24.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-09-15T05:26:25.478 INFO:tasks.workunit.client.0.smithi138.stderr:norecover is unset 2024-09-15T05:26:25.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-09-15T05:26:25.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:26:25.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:26:25.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:26:25.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:26:25.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-09-15T05:26:26.096 INFO:tasks.workunit.client.0.smithi138.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-15T05:26:26.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-09-15T05:26:36.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-09-15T05:26:36.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:26:36.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:26:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:26:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:26:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:26:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:26:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:26:36.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836508 2024-09-15T05:26:36.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836508 2024-09-15T05:26:36.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508' 2024-09-15T05:26:36.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:36.585 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:26:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672980 2024-09-15T05:26:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672980 2024-09-15T05:26:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672980' 2024-09-15T05:26:36.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:26:36.689 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:26:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509456 2024-09-15T05:26:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509456 2024-09-15T05:26:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836508 1-42949672980 2-64424509456' 2024-09-15T05:26:36.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836508 2024-09-15T05:26:36.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:36.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:26:36.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836508 2024-09-15T05:26:36.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:36.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836508 2024-09-15T05:26:36.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-09-15T05:26:36.798 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836508 2024-09-15T05:26:36.798 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:26:37.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836508 2024-09-15T05:26:37.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:26:38.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:26:38.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:26:38.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836508 2024-09-15T05:26:38.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:38.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672980 2024-09-15T05:26:38.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:38.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:26:38.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672980 2024-09-15T05:26:38.568 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:38.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672980 2024-09-15T05:26:38.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-15T05:26:38.569 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672980 2024-09-15T05:26:38.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:26:38.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672981 -lt 42949672980 2024-09-15T05:26:38.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:26:38.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509456 2024-09-15T05:26:38.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:26:38.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:26:38.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509456 2024-09-15T05:26:38.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:26:38.920 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509456 2024-09-15T05:26:38.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509456 2024-09-15T05:26:38.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-15T05:26:38.920 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:26:39.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-15T05:26:39.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-09-15T05:26:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-09-15T05:26:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-09-15T05:26:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-09-15T05:26:39.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-09-15T05:26:39.602 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:26:39.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-09-15T05:26:39.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-09-15T05:26:39.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-09-15T05:26:39.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-09-15T05:26:39.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-09-15T05:26:39.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:26:39.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-09-15T05:26:39.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-09-15T05:26:39.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-09-15T05:26:39.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-09-15T05:26:39.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-09-15T05:26:39.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-09-15T05:26:39.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-09-15T05:26:39.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.967 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-09-15T05:26:39.968 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-09-15T05:26:39.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-09-15T05:26:39.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-09-15T05:26:39.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:26:39.977 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.977 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-09-15T05:26:39.978 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-09-15T05:26:39.978 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-09-15T05:26:39.978 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:26:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-09-15T05:26:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-09-15T05:26:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-09-15T05:26:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:26:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-09-15T05:26:39.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:26:39.983 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:26:39.983 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.983 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-09-15T05:26:39.983 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.990 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-09-15T05:26:39.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-09-15T05:26:39.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-09-15T05:26:39.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:26:39.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-09-15T05:26:39.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-09-15T05:26:40.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-09-15T05:26:40.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:26:40.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:26:40.602 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:26:40.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-09-15T05:26:40.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:26:40.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:26:40.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:26:40.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:26:40.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:26:41.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:26:41.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-09-15T05:26:41.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:26:41.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:26:41.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:26:41.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:26:41.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:26:41.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:26:41.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:26:41.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:26:41.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:26:41.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:41.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:26:41.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:26:41.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:41.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:26:41.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:26:41.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:26:41.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:26:41.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:41.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:26:41.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-recovery-stats 0 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T05:26:41.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-09-15T05:26:41.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:26:41.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:26:41.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:26:41.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:26:41.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:26:41.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:26:41.961 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:26:41.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:26:41.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:26:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:26:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:26:41.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:26:41.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:41.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:26:41.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:26:41.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:41.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:26:41.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T05:26:41.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-09-15T05:26:41.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:26:41.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:41.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.290662 2024-09-15T05:26:41.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.290662 2024-09-15T05:26:41.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:26:41.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:26:41.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T05:26:41.970 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T05:26:41.970 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T05:26:41.997 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-09-15T05:26:41.997 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-09-15T05:26:42.060 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T05:26:42.065 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T05:26:42.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-09-15T05:26:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-09-15T05:26:42.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-09-15T05:26:42.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none ' 2024-09-15T05:26:42.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-09-15T05:26:42.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-09-15T05:26:42.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_rados_repair_warning 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_rep_backfill_unfound 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_rep_read_unfound 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_rep_recovery_unfound' 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:26:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:26:42.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:26:42.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:26:42.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:26:42.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:26:42.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:26:42.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:26:42.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:26:42.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:42.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:26:42.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:26:42.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:26:42.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:26:42.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:26:42.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:26:42.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:26:42.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:42.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:42.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.400988 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-15T05:26:42.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-15T05:26:42.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:26:42.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:26:42.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:26:42.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:26:42.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:42.423 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:42.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:26:42.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-15T05:26:42.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:26:42.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:26:42.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:26:42.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:26:42.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:26:42.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get fsid 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:26:42.733 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:26:42.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get mon_host 2024-09-15T05:26:42.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-15T05:26:42.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:42.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:26:42.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:26:42.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:26:42.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-15T05:26:42.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:26:43.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:26:43.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:26:43.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:26:43.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-15T05:26:43.920 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:26:43.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-15T05:26:43.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:26:43.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:26:43.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:26:43.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43a7155a-651e-448f-8ff4-04fd658c33db 2024-09-15T05:26:43.938 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 43a7155a-651e-448f-8ff4-04fd658c33db 2024-09-15T05:26:43.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 43a7155a-651e-448f-8ff4-04fd658c33db' 2024-09-15T05:26:43.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:26:43.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATcOZmafhuOBAAZsNVgTCf6cef6nTI3TTdYw== 2024-09-15T05:26:43.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATcOZmafhuOBAAZsNVgTCf6cef6nTI3TTdYw=="}' 2024-09-15T05:26:43.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43a7155a-651e-448f-8ff4-04fd658c33db -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:26:44.257 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:26:44.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:26:44.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATcOZmafhuOBAAZsNVgTCf6cef6nTI3TTdYw== --osd-uuid 43a7155a-651e-448f-8ff4-04fd658c33db 2024-09-15T05:26:44.289 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.289+0000 7f43c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:44.303 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.302+0000 7f43c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:44.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.303+0000 7f43c891f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:44.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.304+0000 7f43c891f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:26:44.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.304+0000 7f43c891f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:26:44.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.304+0000 7f43c891f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:26:44.344 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:44.342+0000 7f43c891f440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-15T05:26:49.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-15T05:26:49.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:26:49.670 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:26:49.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:26:49.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:26:50.149 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:26:50.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:26:50.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:26:50.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:26:50.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:26:50.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:26:50.170 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:50.168+0000 7f9b7a221440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:50.172 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:50.170+0000 7f9b7a221440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:50.173 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:50.171+0000 7f9b7a221440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:26:50.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:26:50.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:26:50.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:26:50.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:50.687 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:50.685+0000 7f9b7a221440 -1 Falling back to public interface 2024-09-15T05:26:50.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:51.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:51.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:51.887 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:26:51.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:26:51.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:51.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:52.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:53.226 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:26:53.224+0000 7f9b7a221440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:26:53.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:53.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:53.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:26:53.250 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:26:53.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:53.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:53.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:54.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:54.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:54.605 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:26:54.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:26:54.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:54.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:55.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:56.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:56.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:56.066 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:26:56.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:26:56.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:56.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:56.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:57.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:57.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:57.431 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:26:57.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:26:57.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:57.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:26:58.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:26:58.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:26:58.907 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:26:58.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:26:58.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:26:58.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:26:59.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:00.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:00.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:00.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:27:00.322 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:27:00.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:00.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:00.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:01.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:01.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:01.683 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:27:01.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:27:01.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:01.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:02.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:03.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:03.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:03.198 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:27:03.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:27:03.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:03.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:03.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:04.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:04.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:04.605 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:27:04.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:27:04.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:04.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:04.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:05.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:05.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:05.986 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:27:05.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:27:05.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:05.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:06.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:07.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:07.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:07.561 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:27:07.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:27:07.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:07.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:08.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:09.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:09.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:09.217 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:27:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:27:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:09.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:09.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:10.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:10.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:10.673 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:27:10.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:27:10.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:10.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:11.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:12.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:12.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:12.431 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:27:12.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:27:12.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:12.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:13.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:14.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:14.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:14.075 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:27:14.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:27:14.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:14.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:14.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:15.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:15.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:15.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:27:15.911 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:27:15.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:15.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:27:16.335 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 6 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/402106,v1:127.0.0.1:6803/402106] [v2:127.0.0.1:6804/402106,v1:127.0.0.1:6805/402106] exists,up 43a7155a-651e-448f-8ff4-04fd658c33db 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:27:16.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:27:16.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:27:16.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f342c1dd-ae60-4193-b56b-38994fe444e8 2024-09-15T05:27:16.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f342c1dd-ae60-4193-b56b-38994fe444e8' 2024-09-15T05:27:16.342 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 f342c1dd-ae60-4193-b56b-38994fe444e8 2024-09-15T05:27:16.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:27:16.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0cOZmm6zmFBAAaUFTIFrQbjWBRW4UO82cJg== 2024-09-15T05:27:16.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0cOZmm6zmFBAAaUFTIFrQbjWBRW4UO82cJg=="}' 2024-09-15T05:27:16.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f342c1dd-ae60-4193-b56b-38994fe444e8 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:27:17.118 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:27:17.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:27:17.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0cOZmm6zmFBAAaUFTIFrQbjWBRW4UO82cJg== --osd-uuid f342c1dd-ae60-4193-b56b-38994fe444e8 2024-09-15T05:27:17.153 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.152+0000 7fd3e023b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.201+0000 7fd3e023b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.202+0000 7fd3e023b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.202+0000 7fd3e023b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:27:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.202+0000 7fd3e023b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:27:17.203 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.202+0000 7fd3e023b440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:27:17.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:17.268+0000 7fd3e023b440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-15T05:27:23.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-15T05:27:23.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:27:23.363 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:27:23.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:27:23.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:27:23.850 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:27:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:27:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:27:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:27:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:27:23.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:27:23.871 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:23.871+0000 7fab05211440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:23.873 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:23.872+0000 7fab05211440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:23.876 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:23.874+0000 7fab05211440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:24.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:27:24.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:27:24.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:27:24.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:27:24.244 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:27:24.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:27:24.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:24.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:27:24.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:24.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:24.393 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:24.392+0000 7fab05211440 -1 Falling back to public interface 2024-09-15T05:27:24.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:25.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:25.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:25.595 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:27:25.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:27:25.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:25.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:25.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:26.861 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:26.860+0000 7fab05211440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:27:26.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:26.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:26.953 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:27:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:27:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:26.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:27.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:28.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:28.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:28.531 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:27:28.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:27:28.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:28.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:28.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:29.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:29.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:29.979 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:27:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:27:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:30.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:31.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:31.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:31.435 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:27:31.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:27:31.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:31.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:31.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:32.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:32.834 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:27:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:27:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:33.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:34.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:34.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:34.207 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:27:34.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:27:34.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:34.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:34.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:35.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:35.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:35.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:27:35.678 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:27:35.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:35.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:36.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:37.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:37.230 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:27:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:27:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:37.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:38.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:38.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:38.702 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:27:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:27:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:38.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:39.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:40.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:40.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:40.077 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:27:40.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:27:40.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:40.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:40.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:41.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:41.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:41.658 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:27:41.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:27:41.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:41.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:43.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:43.635 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:27:43.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:27:43.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:43.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:44.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:45.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:45.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:45.461 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:27:45.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:27:45.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:45.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:46.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:47.533 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:27:47.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:27:47.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:47.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:48.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:49.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:49.253 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:27:49.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:27:49.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:49.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:50.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:27:51.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:27:51.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:51.129 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:27:51.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:27:51.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:51.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:27:51.635 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/403861,v1:127.0.0.1:6811/403861] [v2:127.0.0.1:6812/403861,v1:127.0.0.1:6813/403861] exists,up f342c1dd-ae60-4193-b56b-38994fe444e8 2024-09-15T05:27:51.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:27:51.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:27:51.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:27:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=631d2da5-d4e0-4f26-9804-f3fee3b585e1 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 631d2da5-d4e0-4f26-9804-f3fee3b585e1 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 631d2da5-d4e0-4f26-9804-f3fee3b585e1' 2024-09-15T05:27:51.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:27:51.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBXcOZmoCjYJhAAXwI5ZDiIN1fPQB8jm+wJzw== 2024-09-15T05:27:51.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBXcOZmoCjYJhAAXwI5ZDiIN1fPQB8jm+wJzw=="}' 2024-09-15T05:27:51.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 631d2da5-d4e0-4f26-9804-f3fee3b585e1 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:27:52.145 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:27:52.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:27:52.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBXcOZmoCjYJhAAXwI5ZDiIN1fPQB8jm+wJzw== --osd-uuid 631d2da5-d4e0-4f26-9804-f3fee3b585e1 2024-09-15T05:27:52.179 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.178+0000 7fe793b2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.180+0000 7fe793b2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:52.182 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.181+0000 7fe793b2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:52.182 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.181+0000 7fe793b2d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:27:52.182 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.181+0000 7fe793b2d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:27:52.182 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.181+0000 7fe793b2d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:27:52.235 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:52.234+0000 7fe793b2d440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-15T05:27:58.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-15T05:27:58.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:27:58.175 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:27:58.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:27:58.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:27:58.656 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:27:58.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:27:58.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:27:58.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:27:58.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:27:58.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:27:58.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:58.676+0000 7f061ea8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:58.679 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:58.678+0000 7f061ea8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:58.680 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:58.679+0000 7f061ea8d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:27:59.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:27:59.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:27:59.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:27:59.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:27:59.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:27:59.006 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:27:59.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:27:59.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:27:59.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:27:59.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:27:59.196 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:27:59.196+0000 7f061ea8d440 -1 Falling back to public interface 2024-09-15T05:27:59.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:00.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:00.357 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:28:00.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:00.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:28:00.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:00.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:00.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:01.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:01.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:01.710 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:28:01.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:28:01.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:01.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:01.765 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:01.764+0000 7f061ea8d440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:28:02.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:03.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:03.141 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:28:03.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:03.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:28:03.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:03.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:03.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:04.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:04.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:04.680 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:28:04.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:28:04.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:04.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:05.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:06.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:06.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:06.069 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:28:06.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:28:06.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:06.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:06.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:07.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:07.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:07.432 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:28:07.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:28:07.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:07.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:08.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:09.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:09.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:09.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:28:09.110 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:28:09.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:09.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:09.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:10.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:10.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:10.466 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:28:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:28:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:10.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:10.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:11.893 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:28:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:11.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:28:11.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:11.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:12.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:13.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:13.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:13.292 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:28:13.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:28:13.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:13.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:13.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:14.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:14.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:14.783 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:28:14.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:28:14.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:14.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:15.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:16.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:16.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:16.251 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:28:16.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:28:16.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:16.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:17.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:18.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:18.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:18.054 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:28:18.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:28:18.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:18.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:18.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:19.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:19.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:19.840 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:28:19.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:28:19.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:19.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:21.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:22.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:22.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:22.062 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:28:22.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:28:22.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:22.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:22.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:23.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:23.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:23.468 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:28:23.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:28:23.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:23.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:24.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:25.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:25.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:25.164 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:28:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:28:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:26.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:26.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:26.620 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:28:26.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:28:26.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:26.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:28:27.003 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/405616,v1:127.0.0.1:6819/405616] [v2:127.0.0.1:6820/405616,v1:127.0.0.1:6821/405616] exists,up 631d2da5-d4e0-4f26-9804-f3fee3b585e1 2024-09-15T05:28:27.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:28:27.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:28:27.009 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:28:27.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c466307c-7bd7-48a1-b6aa-42c6b6a69949 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 c466307c-7bd7-48a1-b6aa-42c6b6a69949 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 c466307c-7bd7-48a1-b6aa-42c6b6a69949' 2024-09-15T05:28:27.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:28:27.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB7cOZmK+ooARAA1+pvhfPoiUZKqdKMWHPmFg== 2024-09-15T05:28:27.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB7cOZmK+ooARAA1+pvhfPoiUZKqdKMWHPmFg=="}' 2024-09-15T05:28:27.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c466307c-7bd7-48a1-b6aa-42c6b6a69949 -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-15T05:28:28.120 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:28:28.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-15T05:28:28.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB7cOZmK+ooARAA1+pvhfPoiUZKqdKMWHPmFg== --osd-uuid c466307c-7bd7-48a1-b6aa-42c6b6a69949 2024-09-15T05:28:28.153 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.151+0000 7f23cf212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:28.155 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.153+0000 7f23cf212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:28.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.154+0000 7f23cf212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:28.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.154+0000 7f23cf212440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:28:28.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.154+0000 7f23cf212440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:28:28.156 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.154+0000 7f23cf212440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:28:28.268 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:28.267+0000 7f23cf212440 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-15T05:28:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-15T05:28:33.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:28:33.879 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:28:33.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:28:33.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:28:34.357 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:28:34.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:28:34.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:28:34.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:28:34.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:28:34.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:28:34.378 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:34.377+0000 7fa5b1d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:34.380 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:34.379+0000 7fa5b1d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:34.381 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:34.380+0000 7fa5b1d1f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:34.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:28:34.703 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:28:34.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:34.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:34.898 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:34.897+0000 7fa5b1d1f440 -1 Falling back to public interface 2024-09-15T05:28:35.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:36.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:36.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:36.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:28:36.066 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:28:36.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:36.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:36.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:37.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:37.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:37.424 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:28:37.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:28:37.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:37.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:37.431 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:28:37.430+0000 7fa5b1d1f440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:28:37.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:38.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:38.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:38.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:28:38.907 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:28:38.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:38.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:39.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:40.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:40.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:40.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:28:40.364 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:28:40.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:40.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:40.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:41.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:41.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:41.818 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:28:41.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:28:41.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:41.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:42.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:43.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:43.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:43.368 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:28:43.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:28:43.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:43.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:43.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:44.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:44.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:44.776 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:28:44.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:28:44.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:44.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:45.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:46.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:46.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:46.220 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:28:46.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:28:46.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:46.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:46.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:47.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:47.663 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:28:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:28:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:47.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:48.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:49.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:49.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:49.267 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:28:49.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:28:49.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:49.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:49.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:50.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:50.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:50.891 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:28:50.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:28:50.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:50.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:51.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:52.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:52.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:52.318 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:28:52.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:28:52.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:52.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:52.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:53.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:53.937 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:28:53.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:28:53.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:53.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:54.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:55.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:55.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:55.683 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:28:55.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:28:55.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:55.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:56.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:57.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:57.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:57.577 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:28:57.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:28:57.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:57.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:28:58.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:28:59.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:28:59.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:28:59.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:28:59.491 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:28:59.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:28:59.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:29:00.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:29:01.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:29:01.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:29:01.449 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:29:01.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:29:01.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:29:01.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:29:02.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:29:03.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:29:03.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:29:03.205 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:29:03.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:29:03.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:29:03.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:29:03.594 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/407395,v1:127.0.0.1:6827/407395] [v2:127.0.0.1:6828/407395,v1:127.0.0.1:6829/407395] exists,up c466307c-7bd7-48a1-b6aa-42c6b6a69949 2024-09-15T05:29:03.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:03.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:03.680 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:04.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:04.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:04.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803804 2024-09-15T05:29:04.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803804 2024-09-15T05:29:04.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803804' 2024-09-15T05:29:04.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:04.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574866 2024-09-15T05:29:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574866 2024-09-15T05:29:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803804 1-55834574866' 2024-09-15T05:29:04.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:04.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:04.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345932 2024-09-15T05:29:04.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345932 2024-09-15T05:29:04.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803804 1-55834574866 2-85899345932' 2024-09-15T05:29:04.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:04.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:04.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920774 2024-09-15T05:29:04.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920774 2024-09-15T05:29:04.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803804 1-55834574866 2-85899345932 3-141733920774' 2024-09-15T05:29:04.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:04.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803804 2024-09-15T05:29:04.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:04.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:04.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803804 2024-09-15T05:29:04.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:04.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803804 2024-09-15T05:29:04.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803804' 2024-09-15T05:29:04.498 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803804 2024-09-15T05:29:04.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803802 -lt 25769803804 2024-09-15T05:29:04.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:05.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:05.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:06.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803804 -lt 25769803804 2024-09-15T05:29:06.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:06.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574866 2024-09-15T05:29:06.198 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:06.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:06.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574866 2024-09-15T05:29:06.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:06.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574866 2024-09-15T05:29:06.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574866' 2024-09-15T05:29:06.201 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574866 2024-09-15T05:29:06.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:06.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574866 -lt 55834574866 2024-09-15T05:29:06.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:06.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345932 2024-09-15T05:29:06.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:06.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:06.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345932 2024-09-15T05:29:06.561 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345932 2024-09-15T05:29:06.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-09-15T05:29:06.562 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345932 2024-09-15T05:29:06.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:06.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-15T05:29:06.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:06.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920774 2024-09-15T05:29:06.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:06.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:06.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920774 2024-09-15T05:29:06.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:06.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920774 2024-09-15T05:29:06.919 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920774 2024-09-15T05:29:06.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920774' 2024-09-15T05:29:06.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:07.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920774 -lt 141733920774 2024-09-15T05:29:07.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:07.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:07.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:07.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:07.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:08.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:08.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:08.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:08.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:08.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T05:29:08.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:29:08.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:29:08.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-09-15T05:29:08.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-09-15T05:29:08.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-15T05:29:09.662 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'pool-rep' already exists 2024-09-15T05:29:09.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:29:10.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-09-15T05:29:10.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:10.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:10.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:10.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:10.675 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:10.676 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:10.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:10.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:10.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:10.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:10.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:10.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:10.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:10.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:10.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:11.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:11.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:11.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803807 2024-09-15T05:29:11.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803807 2024-09-15T05:29:11.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807' 2024-09-15T05:29:11.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:11.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:11.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574870 2024-09-15T05:29:11.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574870 2024-09-15T05:29:11.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574870' 2024-09-15T05:29:11.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:11.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:11.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345936 2024-09-15T05:29:11.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345936 2024-09-15T05:29:11.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574870 2-85899345936' 2024-09-15T05:29:11.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:11.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920777 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920777 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803807 1-55834574870 2-85899345936 3-141733920777' 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803807 2024-09-15T05:29:11.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:11.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:11.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803807 2024-09-15T05:29:11.964 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:11.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803807 2024-09-15T05:29:11.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-09-15T05:29:11.966 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803807 2024-09-15T05:29:11.966 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:12.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803805 -lt 25769803807 2024-09-15T05:29:12.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:13.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:13.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:13.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803807 2024-09-15T05:29:13.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:13.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574870 2024-09-15T05:29:13.669 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:13.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:13.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574870 2024-09-15T05:29:13.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:13.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574870 2024-09-15T05:29:13.672 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574870 2024-09-15T05:29:13.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-09-15T05:29:13.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:14.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574870 -lt 55834574870 2024-09-15T05:29:14.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:14.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345936 2024-09-15T05:29:14.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:14.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:14.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345936 2024-09-15T05:29:14.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:14.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345936 2024-09-15T05:29:14.022 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345936 2024-09-15T05:29:14.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-09-15T05:29:14.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:14.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345936 -lt 85899345936 2024-09-15T05:29:14.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:14.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920777 2024-09-15T05:29:14.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:14.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:14.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920777 2024-09-15T05:29:14.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:14.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920777 2024-09-15T05:29:14.388 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920777 2024-09-15T05:29:14.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920777' 2024-09-15T05:29:14.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:14.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920777 -lt 141733920777 2024-09-15T05:29:14.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:14.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:14.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:15.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:29:15.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:15.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:15.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:29:15.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:15.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:15.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:16.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:29:16.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:29:16.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-400988 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-400988 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-400988 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:16.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:29:16.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-09-15T05:29:16.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-400988 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-400988 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:29:16.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-400988 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 0 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:16.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:16.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:16.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:16.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:29:16.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:16.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:29:16.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:16.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:16.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:16.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:16.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:16.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:16.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:29:16.994 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:16.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-400988 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-400988 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:29:17.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:29:17.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-400988 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:17.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:17.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:17.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-09-15T05:29:17.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:17.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:17.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:17.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:17.117 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:17.118 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:17.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:17.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:17.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:17.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:17.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:17.595 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:17.595 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:17.595 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:17.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:17.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:17.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:17.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803811 2024-09-15T05:29:17.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803811 2024-09-15T05:29:17.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811' 2024-09-15T05:29:17.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:17.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:17.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574873 2024-09-15T05:29:17.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574873 2024-09-15T05:29:17.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574873' 2024-09-15T05:29:17.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:17.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:17.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-15T05:29:17.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-15T05:29:17.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-85899345939' 2024-09-15T05:29:17.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:17.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:18.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920780 2024-09-15T05:29:18.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920780 2024-09-15T05:29:18.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803811 1-55834574873 2-85899345939 3-141733920780' 2024-09-15T05:29:18.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:18.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803811 2024-09-15T05:29:18.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:18.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:18.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803811 2024-09-15T05:29:18.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:18.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803811 2024-09-15T05:29:18.066 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803811 2024-09-15T05:29:18.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803811' 2024-09-15T05:29:18.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:18.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803809 -lt 25769803811 2024-09-15T05:29:18.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:19.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:19.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:19.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803811 -lt 25769803811 2024-09-15T05:29:19.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:19.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574873 2024-09-15T05:29:19.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:19.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:19.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574873 2024-09-15T05:29:19.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:19.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574873 2024-09-15T05:29:19.774 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574873 2024-09-15T05:29:19.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574873' 2024-09-15T05:29:19.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:20.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574873 -lt 55834574873 2024-09-15T05:29:20.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:20.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345939 2024-09-15T05:29:20.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:20.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:20.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345939 2024-09-15T05:29:20.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:20.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-15T05:29:20.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-15T05:29:20.128 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345939 2024-09-15T05:29:20.129 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:20.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-15T05:29:20.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:20.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920780 2024-09-15T05:29:20.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:20.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:20.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920780 2024-09-15T05:29:20.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:20.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920780 2024-09-15T05:29:20.475 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920780 2024-09-15T05:29:20.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920780' 2024-09-15T05:29:20.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:20.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920781 -lt 141733920780 2024-09-15T05:29:20.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:20.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:20.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:21.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:29:21.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:21.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:21.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:21.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:21.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:21.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:21.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:21.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:29:21.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:21.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:21.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:22.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:29:22.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:29:22.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:29:22.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-09-15T05:29:22.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T05:29:22.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-09-15T05:29:22.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-09-15T05:29:22.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-09-15T05:29:22.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:22.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:22.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:22.565 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:22.565 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:22.566 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:22.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:22.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:22.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:22.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803814 2024-09-15T05:29:22.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803814 2024-09-15T05:29:22.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814' 2024-09-15T05:29:22.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:22.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:22.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574876 2024-09-15T05:29:22.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574876 2024-09-15T05:29:22.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574876' 2024-09-15T05:29:22.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:22.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:22.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345942 2024-09-15T05:29:22.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345942 2024-09-15T05:29:22.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574876 2-85899345942' 2024-09-15T05:29:22.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:22.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:23.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920783 2024-09-15T05:29:23.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920783 2024-09-15T05:29:23.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803814 1-55834574876 2-85899345942 3-141733920783' 2024-09-15T05:29:23.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:23.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803814 2024-09-15T05:29:23.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:23.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:23.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803814 2024-09-15T05:29:23.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:23.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803814 2024-09-15T05:29:23.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803814' 2024-09-15T05:29:23.004 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803814 2024-09-15T05:29:23.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:23.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803814 -lt 25769803814 2024-09-15T05:29:23.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:23.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574876 2024-09-15T05:29:23.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:23.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:23.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574876 2024-09-15T05:29:23.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:23.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574876 2024-09-15T05:29:23.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574876' 2024-09-15T05:29:23.349 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574876 2024-09-15T05:29:23.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:23.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574876 -lt 55834574876 2024-09-15T05:29:23.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:23.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345942 2024-09-15T05:29:23.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:23.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:23.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345942 2024-09-15T05:29:23.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:23.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345942 2024-09-15T05:29:23.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345942' 2024-09-15T05:29:23.705 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345942 2024-09-15T05:29:23.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:24.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345942 -lt 85899345942 2024-09-15T05:29:24.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:24.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920783 2024-09-15T05:29:24.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:24.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:24.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920783 2024-09-15T05:29:24.056 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920783 2024-09-15T05:29:24.057 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920783 2024-09-15T05:29:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920783' 2024-09-15T05:29:24.057 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:24.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920783 -lt 141733920783 2024-09-15T05:29:24.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-09-15T05:29:24.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T05:29:24.761 INFO:tasks.workunit.client.0.smithi138.stderr:dumped all 2024-09-15T05:29:24.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-09-15T05:29:24.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-09-15T05:29:24.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:24.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:24.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:24.773 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:24.773 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:25.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:25.117 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:25.117 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 0 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:25.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:25.119 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:25.489 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:25.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:25.492 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:25.493 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:25.493 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:25.493 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:25.493 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:25.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:25.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:25.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:25.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:25.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:25.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 1 2024-09-15T05:29:25.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:29:25.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:25.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:25.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:25.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:25.656 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:25.656 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:26.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:26.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:29:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:29:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:29:26.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:29:26.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:26.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:26.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:26.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:26.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:29:26.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:26.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:29:26.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:29:26.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:26.184 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:26.185 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:26.185 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:26.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:29:26.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:26.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-400988 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-400988 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:29:26.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-400988 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:26.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:26.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-09-15T05:29:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:26.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:26.290 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:26.291 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:26.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:26.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:26.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:26.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:26.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:26.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803817 2024-09-15T05:29:26.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803817 2024-09-15T05:29:26.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817' 2024-09-15T05:29:26.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:26.828 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574879 2024-09-15T05:29:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574879 2024-09-15T05:29:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-55834574879' 2024-09-15T05:29:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:26.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:27.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345945 2024-09-15T05:29:27.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345945 2024-09-15T05:29:27.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-55834574879 2-85899345945' 2024-09-15T05:29:27.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:27.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920786 2024-09-15T05:29:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920786 2024-09-15T05:29:27.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803817 1-55834574879 2-85899345945 3-141733920786' 2024-09-15T05:29:27.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:27.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803817 2024-09-15T05:29:27.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:27.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:27.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803817 2024-09-15T05:29:27.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:27.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803817 2024-09-15T05:29:27.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-09-15T05:29:27.153 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803817 2024-09-15T05:29:27.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:27.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803817 -lt 25769803817 2024-09-15T05:29:27.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:27.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574879 2024-09-15T05:29:27.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:27.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574879 2024-09-15T05:29:27.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574879 2024-09-15T05:29:27.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-09-15T05:29:27.526 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574879 2024-09-15T05:29:27.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:27.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574879 -lt 55834574879 2024-09-15T05:29:27.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:27.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345945 2024-09-15T05:29:27.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:27.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:27.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345945 2024-09-15T05:29:27.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:27.879 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345945 2024-09-15T05:29:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345945 2024-09-15T05:29:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-09-15T05:29:27.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:28.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345945 -lt 85899345945 2024-09-15T05:29:28.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:28.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920786 2024-09-15T05:29:28.239 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:28.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:28.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920786 2024-09-15T05:29:28.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:28.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920786 2024-09-15T05:29:28.241 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920786 2024-09-15T05:29:28.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920786' 2024-09-15T05:29:28.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:28.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920786 -lt 141733920786 2024-09-15T05:29:28.588 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:28.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:28.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:29.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:29:29.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:29.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:29.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:29.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:29.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:29.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:29.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:29.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:29:29.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:29.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:29.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 0 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:29.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:29.875 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:30.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:30.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:29:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:29:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:29:30.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:30.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:30.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:30.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:30.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:30.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:30.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 2 2024-09-15T05:29:30.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:30.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:30.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:30.389 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:30.389 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:30.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:30.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.0.asok 2024-09-15T05:29:30.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:30.823 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.0.asok 2024-09-15T05:29:30.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:30.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.0.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-400988 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-400988 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:29:30.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-400988 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:30.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:30.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:30.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-09-15T05:29:30.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:30.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:30.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:30.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:31.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:31.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:31.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:31.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:31.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:31.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:31.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:31.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803819 2024-09-15T05:29:31.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803819 2024-09-15T05:29:31.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819' 2024-09-15T05:29:31.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:31.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:31.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574882 2024-09-15T05:29:31.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574882 2024-09-15T05:29:31.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-55834574882' 2024-09-15T05:29:31.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:31.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:31.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345948 2024-09-15T05:29:31.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345948 2024-09-15T05:29:31.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-55834574882 2-85899345948' 2024-09-15T05:29:31.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:31.743 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:31.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920789 2024-09-15T05:29:31.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920789 2024-09-15T05:29:31.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803819 1-55834574882 2-85899345948 3-141733920789' 2024-09-15T05:29:31.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:31.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803819 2024-09-15T05:29:31.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:31.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:31.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803819 2024-09-15T05:29:31.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:31.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803819 2024-09-15T05:29:31.877 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803819 2024-09-15T05:29:31.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803819' 2024-09-15T05:29:31.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:32.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803817 -lt 25769803819 2024-09-15T05:29:32.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:33.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:33.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:33.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803820 -lt 25769803819 2024-09-15T05:29:33.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:33.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574882 2024-09-15T05:29:33.574 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:33.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:33.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574882 2024-09-15T05:29:33.576 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:33.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574882 2024-09-15T05:29:33.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574882' 2024-09-15T05:29:33.577 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574882 2024-09-15T05:29:33.577 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:33.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574882 -lt 55834574882 2024-09-15T05:29:33.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:33.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345948 2024-09-15T05:29:33.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:33.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:33.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345948 2024-09-15T05:29:33.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:33.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345948 2024-09-15T05:29:33.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345948' 2024-09-15T05:29:33.933 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345948 2024-09-15T05:29:33.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:34.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345948 -lt 85899345948 2024-09-15T05:29:34.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:34.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920789 2024-09-15T05:29:34.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:34.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:34.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920789 2024-09-15T05:29:34.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:34.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920789 2024-09-15T05:29:34.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920789' 2024-09-15T05:29:34.280 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920789 2024-09-15T05:29:34.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:34.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920789 -lt 141733920789 2024-09-15T05:29:34.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:34.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:34.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:35.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:29:35.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:35.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:35.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:35.106 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:35.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:35.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:35.107 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:35.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:29:35.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:35.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:35.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:35.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:29:35.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:29:35.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:29:35.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-09-15T05:29:35.914 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T05:29:36.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-09-15T05:29:36.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-09-15T05:29:36.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-09-15T05:29:36.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:36.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:36.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:36.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:36.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803822 2024-09-15T05:29:36.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803822 2024-09-15T05:29:36.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822' 2024-09-15T05:29:36.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:36.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:36.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574885 2024-09-15T05:29:36.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574885 2024-09-15T05:29:36.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574885' 2024-09-15T05:29:36.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:36.618 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:36.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345951 2024-09-15T05:29:36.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345951 2024-09-15T05:29:36.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574885 2-85899345951' 2024-09-15T05:29:36.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:36.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:36.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920792 2024-09-15T05:29:36.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920792 2024-09-15T05:29:36.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803822 1-55834574885 2-85899345951 3-141733920792' 2024-09-15T05:29:36.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:36.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803822 2024-09-15T05:29:36.824 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:36.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:36.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803822 2024-09-15T05:29:36.825 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:36.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803822 2024-09-15T05:29:36.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803822' 2024-09-15T05:29:36.827 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803822 2024-09-15T05:29:36.827 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:37.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803820 -lt 25769803822 2024-09-15T05:29:37.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:38.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:38.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:38.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803823 -lt 25769803822 2024-09-15T05:29:38.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:38.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574885 2024-09-15T05:29:38.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:38.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:38.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574885 2024-09-15T05:29:38.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:38.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574885 2024-09-15T05:29:38.537 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574885 2024-09-15T05:29:38.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574885' 2024-09-15T05:29:38.537 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:38.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574885 -lt 55834574885 2024-09-15T05:29:38.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:38.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345951 2024-09-15T05:29:38.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:38.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:38.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345951 2024-09-15T05:29:38.892 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:38.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345951 2024-09-15T05:29:38.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345951' 2024-09-15T05:29:38.893 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345951 2024-09-15T05:29:38.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:39.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345951 -lt 85899345951 2024-09-15T05:29:39.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:39.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920792 2024-09-15T05:29:39.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:39.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:39.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920792 2024-09-15T05:29:39.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:39.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920792 2024-09-15T05:29:39.246 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920792 2024-09-15T05:29:39.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920792' 2024-09-15T05:29:39.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:39.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920793 -lt 141733920792 2024-09-15T05:29:39.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-09-15T05:29:39.601 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T05:29:39.942 INFO:tasks.workunit.client.0.smithi138.stderr:dumped all 2024-09-15T05:29:39.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 0 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:39.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:39.956 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:40.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:29:40.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:40.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:40.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:29:40.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:40.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:29:40.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:40.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:29:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:40.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:40.307 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:29:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:40.308 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:40.309 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:40.310 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:40.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:40.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:40.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:40.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 1 2024-09-15T05:29:40.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:40.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:40.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:40.469 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:40.469 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:40.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:40.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:40.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:29:40.859 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:40.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:40.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:40.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:41.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-400988 td/osd-rep-recov-eio.sh 2 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-eio-400988 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=2 2024-09-15T05:29:41.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:29:41.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-eio-400988 2024-09-15T05:29:41.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:29:41.024 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-400988 2024-09-15T05:29:41.024 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-400988 2024-09-15T05:29:41.024 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:29:41.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:29:41.386 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:41.386 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:29:41.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=0 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:29:41.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:41.391 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:41.392 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:41.392 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.0.asok 2024-09-15T05:29:41.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:29:41.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:29:41.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:29:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.0 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.0.asok 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:29:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.0.asok injectdataerr pool-rep obj-eio-400988 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-400988 hang 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-400988 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-09-15T05:29:41.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-400988 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:29:46.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:29:46.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:29:46.555 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:29:46.555 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:29:46.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:29:46.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:29:46.555 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:29:46.650 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:29:47.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:29:47.076 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:29:47.076 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:29:47.076 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:29:47.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:29:47.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:47.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:29:47.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803827 2024-09-15T05:29:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803827 2024-09-15T05:29:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803827' 2024-09-15T05:29:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:47.180 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:29:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574889 2024-09-15T05:29:47.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574889 2024-09-15T05:29:47.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803827 1-55834574889' 2024-09-15T05:29:47.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:47.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:29:47.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345955 2024-09-15T05:29:47.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345955 2024-09-15T05:29:47.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803827 1-55834574889 2-85899345955' 2024-09-15T05:29:47.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:29:47.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:29:47.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920796 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920796 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803827 1-55834574889 2-85899345955 3-141733920796' 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803827 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:47.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:29:47.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803827 2024-09-15T05:29:47.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:47.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803827 2024-09-15T05:29:47.549 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803827 2024-09-15T05:29:47.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-09-15T05:29:47.550 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:29:47.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803827 -lt 25769803827 2024-09-15T05:29:47.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:47.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574889 2024-09-15T05:29:47.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:47.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:29:47.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574889 2024-09-15T05:29:47.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:47.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574889 2024-09-15T05:29:47.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574889' 2024-09-15T05:29:47.898 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574889 2024-09-15T05:29:47.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:48.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574887 -lt 55834574889 2024-09-15T05:29:48.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:29:49.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:29:49.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:29:49.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574889 -lt 55834574889 2024-09-15T05:29:49.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:49.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345955 2024-09-15T05:29:49.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:29:49.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345955 2024-09-15T05:29:49.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345955 2024-09-15T05:29:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345955' 2024-09-15T05:29:49.633 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345955 2024-09-15T05:29:49.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:29:49.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345955 -lt 85899345955 2024-09-15T05:29:49.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:29:49.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920796 2024-09-15T05:29:49.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:29:49.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:29:49.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920796 2024-09-15T05:29:49.993 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:29:49.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920796 2024-09-15T05:29:49.994 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920796 2024-09-15T05:29:49.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920796' 2024-09-15T05:29:49.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:29:50.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920797 -lt 141733920796 2024-09-15T05:29:50.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:29:50.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:50.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:50.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:51.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:51.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:51.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:51.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:29:51.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:29:51.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:29:51.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:51.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:52.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:52.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:52.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:52.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:29:52.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:29:52.532 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:29:53.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:29:53.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:29:53.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T05:29:53.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:29:53.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T05:29:53.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:29:53.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:53.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:53.205 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:53.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:53.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:53.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:53.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:29:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:29:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:29:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:29:54.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:29:54.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:29:54.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:29:54.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:29:54.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:29:54.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:29:54.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:29:54.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-09-15T05:29:54.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:29:54.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:54.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:54.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:55.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:55.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:55.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:55.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:29:55.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:29:56.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:29:56.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:29:56.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-09-15T05:29:56.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:29:56.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-09-15T05:29:56.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:29:56.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:56.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:56.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:56.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:56.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:56.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:56.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:29:57.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:29:57.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:29:57.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:29:57.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:29:57.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:29:57.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:29:57.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:29:57.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:29:57.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:29:57.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:29:57.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:29:57.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:29:57.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:29:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:29:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:29:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-09-15T05:29:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:29:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-09-15T05:29:59.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:29:59.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:29:59.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:00.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:00.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:00.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:00.220 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:00.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:30:01.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:30:01.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:30:01.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-09-15T05:30:01.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:30:01.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-09-15T05:30:04.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:30:04.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:30:04.357 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:04.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:04.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:04.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:04.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:05.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:05.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:30:05.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:30:05.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:30:05.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-09-15T05:30:05.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:30:05.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-09-15T05:30:12.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:30:12.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:30:12.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:30:12.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:30:12.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:30:12.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:30:12.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:30:12.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:12.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:12.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:12.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:12.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:12.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:12.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:30:13.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:30:13.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:30:13.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-09-15T05:30:13.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:30:13.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-09-15T05:30:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:30:26.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:30:26.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:26.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:26.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:26.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:26.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:27.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:27.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:27.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:27.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:27.035 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:30:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:30:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:30:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-09-15T05:30:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:30:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-15T05:30:42.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:30:42.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:30:42.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:30:42.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:30:42.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:30:42.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:30:42.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:30:42.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:42.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:42.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:42.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:42.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:43.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:30:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:30:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:30:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-09-15T05:30:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:30:43.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:30:58.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:30:58.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:30:59.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:30:59.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:30:59.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:30:59.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:30:59.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:30:59.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:30:59.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:31:00.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:31:00.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:31:00.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-09-15T05:31:00.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:31:00.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-15T05:31:15.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:31:15.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:31:15.102 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:31:15.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:31:15.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:31:15.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:31:15.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:31:15.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:31:15.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:31:16.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:31:16.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:31:16.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-09-15T05:31:16.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:31:16.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-09-15T05:31:31.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:31:31.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:31:31.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:31:31.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:31:31.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:31:31.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:31:31.750 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:31:32.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:31:32.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:31:32.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:31:32.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:31:32.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:31:32.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-09-15T05:31:32.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:31:32.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-09-15T05:31:37.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:31:37.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:31:37.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:31:37.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:31:37.207 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:31:37.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:31:37.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:31:37.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:31:37.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:31:37.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:31:37.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:31:37.575 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:31:38.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:31:38.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:31:38.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:31:38.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:31:38.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:31:38.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-09-15T05:31:38.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-15T05:31:38.968 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_fingerprint": "9f1237ac-931c-49d9-8af6-6ce274316070", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "17.2.7-1643-gd75f2382", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "commit": "d75f23820806da622f561ee5ee7551015285993f", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "timestamp": "2024-09-15T05:31:38.967086+0000", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "tag": "", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "health": { 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "status": "HEALTH_ERR", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "checks": { 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "OBJECT_UNFOUND": { 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T05:31:38.969 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1/1 objects unfound (100.000%)", 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DAMAGED": { 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_ERR", 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-09-15T05:31:38.970 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "PG_DEGRADED": { 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-09-15T05:31:38.971 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T05:31:38.972 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "POOL_APP_NOT_ENABLED": { 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "2 pool(s) do not have an application enabled", 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 2 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [ 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'foo'" 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.973 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "SLOW_OPS": { 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "severity": "HEALTH_WARN", 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "summary": { 2024-09-15T05:31:38.974 INFO:tasks.workunit.client.0.smithi138.stdout: "message": "1 slow ops, oldest one blocked for 111 sec, osd.3 has slow ops", 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: "count": 1 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: "detail": [], 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: "muted": false 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: "mutes": [] 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.975 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_first_committed": 1, 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap_last_committed": 1, 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "monmap": { 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "f8d4f6e2-8037-4736-acc3-09b9147ba5da", 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T05:26:42.096366Z", 2024-09-15T05:31:38.976 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T05:26:42.096366Z", 2024-09-15T05:31:38.977 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release": 17, 2024-09-15T05:31:38.977 INFO:tasks.workunit.client.0.smithi138.stdout: "min_mon_release_name": "quincy", 2024-09-15T05:31:38.977 INFO:tasks.workunit.client.0.smithi138.stdout: "election_strategy": 1, 2024-09-15T05:31:38.977 INFO:tasks.workunit.client.0.smithi138.stdout: "disallowed_leaders: ": "", 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": false, 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "tiebreaker_mon": "", 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_ranks: ": "", 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "features": { 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "persistent": [ 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "kraken", 2024-09-15T05:31:38.978 INFO:tasks.workunit.client.0.smithi138.stdout: "luminous", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "mimic", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap-prune", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "nautilus", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "octopus", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "pacific", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "elector-pinging", 2024-09-15T05:31:38.979 INFO:tasks.workunit.client.0.smithi138.stdout: "quincy" 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: "optional": [] 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: "mons": [ 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: "rank": 0, 2024-09-15T05:31:38.980 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "a", 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7140", 2024-09-15T05:31:38.981 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 0 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:7140/0", 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:7140/0", 2024-09-15T05:31:38.982 INFO:tasks.workunit.client.0.smithi138.stdout: "priority": 0, 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 0, 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_location": "{}" 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.983 INFO:tasks.workunit.client.0.smithi138.stdout: "quorum": [ 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: 0 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap": { 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 39, 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: "fsid": "f8d4f6e2-8037-4736-acc3-09b9147ba5da", 2024-09-15T05:31:38.984 INFO:tasks.workunit.client.0.smithi138.stdout: "created": "2024-09-15T05:26:42.946283+0000", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T05:29:10.524031+0000", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "last_up_change": "2024-09-15T05:29:02.995565+0000", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "last_in_change": "2024-09-15T05:28:27.691166+0000", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_num": 5799936, 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_set": [ 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "pglog_hardlimit", 2024-09-15T05:31:38.985 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snapdirs", 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "recovery_deletes", 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "sortbitwise" 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_version": 9, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "full_ratio": 0.99000000953674316, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_snapshot": "", 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_max": 2, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "max_osd": 4, 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "require_min_compat_client": "luminous", 2024-09-15T05:31:38.986 INFO:tasks.workunit.client.0.smithi138.stdout: "min_compat_client": "luminous", 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "require_osd_release": "quincy", 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "pools": [ 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "foo", 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T05:26:43.810194+0000", 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T05:31:38.987 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 8, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 8, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 8, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 8, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 8, 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T05:31:38.988 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T05:31:38.989 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "9", 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T05:31:38.990 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T05:31:38.991 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T05:31:38.992 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 2, 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_name": "pool-rep", 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "create_time": "2024-09-15T05:29:08.875609+0000", 2024-09-15T05:31:38.993 INFO:tasks.workunit.client.0.smithi138.stdout: "flags": 1, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "flags_names": "hashpspool", 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "size": 3, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "min_size": 2, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_rule": 0, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_count": 0, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_target": 0, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_barrier": 0, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-15T05:31:38.994 INFO:tasks.workunit.client.0.smithi138.stdout: "object_hash": 2, 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_autoscale_mode": "off", 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num": 1, 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num": 1, 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_placement_num_target": 1, 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_target": 1, 2024-09-15T05:31:38.995 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_num_pending": 1, 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pg_merge_meta": { 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "source_pgid": "0.0", 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "ready_epoch": 0, 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_started": 0, 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 0, 2024-09-15T05:31:38.996 INFO:tasks.workunit.client.0.smithi138.stdout: "source_version": "0'0", 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "target_version": "0'0" 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "39", 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend": "0", 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "last_force_op_resend_preluminous": "0", 2024-09-15T05:31:38.997 INFO:tasks.workunit.client.0.smithi138.stdout: "auid": 0, 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_mode": "selfmanaged", 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_seq": 0, 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_epoch": 0, 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_snaps": [], 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps": "[]", 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_bytes": 0, 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "quota_max_objects": 0, 2024-09-15T05:31:38.998 INFO:tasks.workunit.client.0.smithi138.stdout: "tiers": [], 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "tier_of": -1, 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "read_tier": -1, 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "write_tier": -1, 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_mode": "none", 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_bytes": 0, 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "target_max_objects": 0, 2024-09-15T05:31:38.999 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_flush_age": 0, 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "cache_min_evict_age": 0, 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profile": "", 2024-09-15T05:31:39.000 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_params": { 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "none" 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_period": 0, 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_count": 0, 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: "use_gmt_hitset": true, 2024-09-15T05:31:39.001 INFO:tasks.workunit.client.0.smithi138.stdout: "min_read_recency_for_promote": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "min_write_recency_for_promote": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_grade_decay_rate": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "hit_set_search_last_n": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "grade_table": [], 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "stripe_width": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "expected_num_objects": 0, 2024-09-15T05:31:39.002 INFO:tasks.workunit.client.0.smithi138.stdout: "fast_read": false, 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: "options": {}, 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: "application_metadata": {} 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: "osds": [ 2024-09-15T05:31:39.003 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "43a7155a-651e-448f-8ff4-04fd658c33db", 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T05:31:39.004 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 6, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 36, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 0, 2024-09-15T05:31:39.005 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6802", 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.006 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6803", 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6804", 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.007 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6805", 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6808", 2024-09-15T05:31:39.008 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6809", 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T05:31:39.009 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6806", 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6807", 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 402106 2024-09-15T05:31:39.010 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6803/402106", 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6805/402106", 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6809/402106", 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6807/402106", 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.011 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "f342c1dd-ae60-4193-b56b-38994fe444e8", 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T05:31:39.012 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 13, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 36, 2024-09-15T05:31:39.013 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 0, 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6810", 2024-09-15T05:31:39.014 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6811", 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.015 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6812", 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.016 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6813", 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.017 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.020 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T05:31:39.020 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.020 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.020 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6816", 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6817", 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.021 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6814", 2024-09-15T05:31:39.022 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6815", 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 403861 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.023 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6811/403861", 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6813/403861", 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6817/403861", 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6815/403861", 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.024 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "631d2da5-d4e0-4f26-9804-f3fee3b585e1", 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T05:31:39.025 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 20, 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 36, 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 0, 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6818", 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.026 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6819", 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6820", 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.027 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6821", 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6824", 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.028 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6825", 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.029 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6822", 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6823", 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 405616 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6819/405616", 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6821/405616", 2024-09-15T05:31:39.030 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6825/405616", 2024-09-15T05:31:39.031 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6823/405616", 2024-09-15T05:31:39.031 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T05:31:39.031 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T05:31:39.031 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T05:31:39.031 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 3, 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: "uuid": "c466307c-7bd7-48a1-b6aa-42c6b6a69949", 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 1, 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: "in": 1, 2024-09-15T05:31:39.032 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 1, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_affinity": 1, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_begin": 0, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_end": 0, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 33, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "up_thru": 36, 2024-09-15T05:31:39.033 INFO:tasks.workunit.client.0.smithi138.stdout: "down_at": 0, 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: "lost_at": 0, 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addrs": { 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.034 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6826", 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6827", 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.035 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addrs": { 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6828", 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6829", 2024-09-15T05:31:39.036 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addrs": { 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6832", 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.037 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6833", 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addrs": { 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "addrvec": [ 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v2", 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6830", 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "v1", 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "addr": "127.0.0.1:6831", 2024-09-15T05:31:39.038 INFO:tasks.workunit.client.0.smithi138.stdout: "nonce": 407395 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "public_addr": "127.0.0.1:6827/407395", 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "cluster_addr": "127.0.0.1:6829/407395", 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_back_addr": "127.0.0.1:6833/407395", 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "heartbeat_front_addr": "127.0.0.1:6831/407395", 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "state": [ 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "exists", 2024-09-15T05:31:39.039 INFO:tasks.workunit.client.0.smithi138.stdout: "up" 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_xinfo": [ 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "0.000000", 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T05:31:39.040 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T05:27:15.700507+0000", 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "0.000000", 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T05:27:50.245970+0000", 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.041 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "0.000000", 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T05:28:25.354583+0000", 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.042 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 3, 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "down_stamp": "0.000000", 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_probability": 0, 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "laggy_interval": 0, 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "features": 4540138320759226367, 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "old_weight": 0, 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "last_purged_snaps_scrub": "2024-09-15T05:29:02.677937+0000", 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: "dead_epoch": 0 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.043 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap": [], 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_upmap_items": [ 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: "pgid": "1.5", 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: "mappings": [ 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.044 INFO:tasks.workunit.client.0.smithi138.stdout: "from": 0, 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: "to": 1 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: "pgid": "1.7", 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: "mappings": [ 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: "from": 0, 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: "to": 2 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.045 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_temp": [], 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "primary_temp": [], 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "blocklist": {}, 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "range_blocklist": {}, 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "erasure_code_profiles": { 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "default": { 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "k": "2", 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "m": "2", 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "plugin": "jerasure", 2024-09-15T05:31:39.046 INFO:tasks.workunit.client.0.smithi138.stdout: "technique": "reed_sol_van" 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "removed_snaps_queue": [], 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "new_removed_snaps": [], 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "new_purged_snaps": [], 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "crush_node_flags": {}, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "device_class_flags": {}, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode": { 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_enabled": false, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_bucket_count": 0, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "degraded_stretch_mode": 0, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "recovering_stretch_mode": 0, 2024-09-15T05:31:39.047 INFO:tasks.workunit.client.0.smithi138.stdout: "stretch_mode_bucket": 0 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_metadata": [ 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6804/402106,v1:127.0.0.1:6805/402106]", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T05:31:39.048 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T05:31:39.049 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T05:26:49.017391Z", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6802/402106,v1:127.0.0.1:6803/402106]", 2024-09-15T05:31:39.050 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/402106,v1:127.0.0.1:6809/402106]", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/402106,v1:127.0.0.1:6807/402106]", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T05:31:39.051 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6812/403861,v1:127.0.0.1:6813/403861]", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T05:31:39.052 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.053 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T05:27:22.710607Z", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6810/403861,v1:127.0.0.1:6811/403861]", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T05:31:39.054 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/403861,v1:127.0.0.1:6817/403861]", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/403861,v1:127.0.0.1:6815/403861]", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T05:31:39.055 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T05:31:39.056 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6820/405616,v1:127.0.0.1:6821/405616]", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T05:31:39.057 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.058 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T05:27:57.506164Z", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T05:31:39.059 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6818/405616,v1:127.0.0.1:6819/405616]", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/405616,v1:127.0.0.1:6825/405616]", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/405616,v1:127.0.0.1:6823/405616]", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T05:31:39.060 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.061 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 3, 2024-09-15T05:31:39.062 INFO:tasks.workunit.client.0.smithi138.stderr:report 805811446 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "arch": "x86_64", 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "back_addr": "[v2:127.0.0.1:6828/407395,v1:127.0.0.1:6829/407395]", 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "back_iface": "", 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs": "1", 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_db": "0", 2024-09-15T05:31:39.063 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_dedicated_wal": "0", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluefs_single_shared_device": "1", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_allocation_from_file": "0", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_access_mode": "file", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_block_size": "4096", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_devices": "sda", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-09-15T05:31:39.064 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_rotational": "1", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_size": "107374182400", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_support_discard": "1", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_bdev_type": "hdd", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "bluestore_min_alloc_size": "4096", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_release": "quincy", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.065 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_short": "17.2.7-1643-gd75f2382", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "ceph_version_when_created": "ceph version 17.2.7-1643-gd75f2382 (d75f23820806da622f561ee5ee7551015285993f) quincy (stable)", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "created_at": "2024-09-15T05:28:33.172238Z", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "default_device_class": "hdd", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "device_ids": "sda=ATA_ST1000NM0033-9ZM173_Z1W5XKK0", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": "sda", 2024-09-15T05:31:39.066 INFO:tasks.workunit.client.0.smithi138.stdout: "distro": "centos", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_description": "CentOS Stream 9", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "distro_version": "9", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "front_addr": "[v2:127.0.0.1:6826/407395,v1:127.0.0.1:6827/407395]", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "front_iface": "", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/407395,v1:127.0.0.1:6833/407395]", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/407395,v1:127.0.0.1:6831/407395]", 2024-09-15T05:31:39.067 INFO:tasks.workunit.client.0.smithi138.stdout: "hostname": "smithi138", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "journal_rotational": "1", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "kernel_version": "5.14.0-505.el9.x86_64", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_swap_kb": "0", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "mem_total_kb": "32487204", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-15T05:31:39.068 INFO:tasks.workunit.client.0.smithi138.stdout: "os": "Linux", 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_objectstore": "bluestore", 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "osdspec_affinity": "", 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "rotational": "1" 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_clean_epochs": { 2024-09-15T05:31:39.069 INFO:tasks.workunit.client.0.smithi138.stdout: "min_last_epoch_clean": 0, 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": { 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: "per_pool": [ 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: "floor": 4294967295 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.070 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_epochs": [ 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 7 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 13 2024-09-15T05:31:39.071 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 20 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 3, 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 33 2024-09-15T05:31:39.072 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_first_committed": 1, 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: "osdmap_last_committed": 39, 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: "crushmap": { 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: "devices": [ 2024-09-15T05:31:39.073 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.0", 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.1", 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.2", 2024-09-15T05:31:39.074 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 3, 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd.3", 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "class": "hdd" 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "types": [ 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 0, 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "osd" 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T05:31:39.075 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "host" 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 2, 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "chassis" 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 3, 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "rack" 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.076 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 4, 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "row" 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 5, 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pdu" 2024-09-15T05:31:39.077 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 6, 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "pod" 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 7, 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "room" 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.078 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 8, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "datacenter" 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 9, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "zone" 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 10, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "region" 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "root" 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.079 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "buckets": [ 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -1, 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default", 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T05:31:39.080 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -2, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "default~hdd", 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 11, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "root", 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.081 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -3, 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138", 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T05:31:39.082 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T05:31:39.083 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 3, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.084 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 3 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "id": -4, 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "name": "smithi138~hdd", 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "type_id": 1, 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "type_name": "host", 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 25608, 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "alg": "straw2", 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "hash": "rjenkins1", 2024-09-15T05:31:39.085 INFO:tasks.workunit.client.0.smithi138.stdout: "items": [ 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 0, 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 0 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 1, 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.086 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 1 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 2, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 2 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "id": 3, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "weight": 6402, 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "pos": 3 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.087 INFO:tasks.workunit.client.0.smithi138.stdout: "rules": [ 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_id": 0, 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "rule_name": "replicated_rule", 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "type": 1, 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "steps": [ 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "take", 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "item": -1, 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "item_name": "default" 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.088 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "choose_firstn", 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 0, 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "type": "osd" 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "op": "emit" 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "tunables": { 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_tries": 0, 2024-09-15T05:31:39.089 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_local_fallback_tries": 0, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_total_tries": 50, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_descend_once": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_vary_r": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "chooseleaf_stable": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "straw_calc_version": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "allowed_bucket_algs": 54, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "profile": "jewel", 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "optimal_tunables": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "legacy_tunables": 0, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "minimum_required_version": "jewel", 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables2": 1, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v2_rules": 0, 2024-09-15T05:31:39.090 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables3": 1, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v3_rules": 0, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v4_buckets": 1, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "require_feature_tunables5": 1, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "has_v5_rules": 0 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "choose_args": {} 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "fsmap": { 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 1, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "default_fscid": -1, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": { 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "compat": {}, 2024-09-15T05:31:39.091 INFO:tasks.workunit.client.0.smithi138.stdout: "ro_compat": {}, 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "incompat": { 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_1": "base v0.20", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_2": "client writeable ranges", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_3": "default file layouts on dirs", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_4": "dir inode in separate object", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_5": "mds uses versioned encoding", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_8": "no anchor table", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_9": "file layout v2", 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_10": "snaprealm v2" 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "feature_flags": { 2024-09-15T05:31:39.092 INFO:tasks.workunit.client.0.smithi138.stdout: "enable_multiple": true, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "ever_enabled_multiple": true 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "standbys": [], 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "filesystems": [] 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_first_committed": 1, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "mdsmap_last_committed": 1, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "auth": { 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 1, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 6, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "num_secrets": 4 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.093 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 9, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_active": 9, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_unknown": 0, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osd": 4, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_sum": { 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 4096, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 1, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 3, 2024-09-15T05:31:39.094 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 3, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 1, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 1, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 3, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 12, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 1, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 4, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T05:31:39.095 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 3, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 12288, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T05:31:39.096 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T05:31:39.097 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 3 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T05:31:39.098 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 1, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 1, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 27, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 27, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 0 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum": { 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T05:31:39.099 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 27, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 4, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 4, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 4, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 419430400, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 1189936, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 1288, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 6, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 1188601, 2024-09-15T05:31:39.100 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 418240464, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 429496729600, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 428278235136, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1318912, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 447406, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 6816, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 1217127776 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T05:31:39.101 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 4, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [ 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.102 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: 1 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 262144 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T05:31:39.103 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_sum_by_class": { 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "hdd": { 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "up_from": 0, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 0, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pgs": 27, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "num_osds": 4, 2024-09-15T05:31:39.104 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_osds": 4, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "num_per_pool_omap_osds": 4, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb": 419430400, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used": 1189936, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_data": 1288, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_omap": 6, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_used_meta": 1188601, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "kb_avail": 418240464, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "statfs": { 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 429496729600, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 428278235136, 2024-09-15T05:31:39.105 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 1318912, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 447406, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 6816, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 1217127776 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "hb_peers": [], 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "snap_trim_queue_len": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "num_snap_trimming": 0, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shards_repaired": 4, 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "op_queue_age_hist": { 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: "histogram": [ 2024-09-15T05:31:39.106 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.107 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: 1 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "upper_bound": 262144 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "perf_stat": { 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ms": 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ms": 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "commit_latency_ns": 0, 2024-09-15T05:31:39.108 INFO:tasks.workunit.client.0.smithi138.stdout: "apply_latency_ns": 0 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "alerts": [], 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "network_ping_times": [] 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "pool_stats": [ 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 2, 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 1, 2024-09-15T05:31:39.109 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 4096, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 1, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 3, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 3, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 1, 2024-09-15T05:31:39.110 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 1, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 3, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 12, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 1, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 4, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 3, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 12288, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T05:31:39.111 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T05:31:39.112 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 3 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T05:31:39.113 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 12288, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 12288, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 1, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 1, 2024-09-15T05:31:39.114 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 3, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 3, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 3 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "poolid": 1, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg": 8, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 0, 2024-09-15T05:31:39.115 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T05:31:39.116 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T05:31:39.117 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T05:31:39.118 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "store_stats": { 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "total": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "available": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "internally_reserved": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "allocated": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "data_stored": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_allocated": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "data_compressed_original": 0, 2024-09-15T05:31:39.119 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_allocated": 0, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "internal_metadata": 0 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 0, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 0, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "up": 24, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": 24, 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "num_store_stats": 4 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: "osd_stats": [ 2024-09-15T05:31:39.120 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 25769803849 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 55834574911 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 85899345977 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.121 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 3, 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "seq": 141733920818 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_state": [ 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 1 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+clean", 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "num": 8 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.122 INFO:tasks.workunit.client.0.smithi138.stdout: "num_pg_by_osd": [ 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 3, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 4, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 7, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 2, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 6, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.123 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 1, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 3, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 7, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "osd": 0, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_primary_pg": 1, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_acting_pg": 7, 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "num_up_not_acting_pg": 0 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.124 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [ 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 1, 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "pool": 2, 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": {} 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "servicemap": { 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "epoch": 6, 2024-09-15T05:31:39.125 INFO:tasks.workunit.client.0.smithi138.stdout: "modified": "2024-09-15T05:30:05.204292+0000", 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "services": {} 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "paxos": { 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "first_committed": 252, 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "last_committed": 790, 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "last_pn": 0, 2024-09-15T05:31:39.126 INFO:tasks.workunit.client.0.smithi138.stdout: "accepted_pn": 0 2024-09-15T05:31:39.127 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:31:39.127 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:31:39.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:31:39.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:31:39.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803851 2024-09-15T05:31:39.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803851 2024-09-15T05:31:39.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851' 2024-09-15T05:31:39.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:31:39.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:31:39.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574913 2024-09-15T05:31:39.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574913 2024-09-15T05:31:39.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574913' 2024-09-15T05:31:39.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:31:39.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:31:39.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345979 2024-09-15T05:31:39.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345979 2024-09-15T05:31:39.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574913 2-85899345979' 2024-09-15T05:31:39.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:31:39.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:31:39.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=141733920821 2024-09-15T05:31:39.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 141733920821 2024-09-15T05:31:39.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803851 1-55834574913 2-85899345979 3-141733920821' 2024-09-15T05:31:39.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:31:39.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803851 2024-09-15T05:31:39.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:31:39.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:31:39.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803851 2024-09-15T05:31:39.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:31:39.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803851 2024-09-15T05:31:39.935 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803851 2024-09-15T05:31:39.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803851' 2024-09-15T05:31:39.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:31:40.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803849 -lt 25769803851 2024-09-15T05:31:40.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:31:41.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:31:41.281 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:31:41.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803851 -lt 25769803851 2024-09-15T05:31:41.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:31:41.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574913 2024-09-15T05:31:41.642 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:31:41.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:31:41.644 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574913 2024-09-15T05:31:41.644 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:31:41.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574913 2024-09-15T05:31:41.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574913' 2024-09-15T05:31:41.645 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574913 2024-09-15T05:31:41.645 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:31:42.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574913 -lt 55834574913 2024-09-15T05:31:42.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:31:42.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345979 2024-09-15T05:31:42.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:31:42.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:31:42.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345979 2024-09-15T05:31:42.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:31:42.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345979 2024-09-15T05:31:42.068 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345979 2024-09-15T05:31:42.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345979' 2024-09-15T05:31:42.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:31:42.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345979 -lt 85899345979 2024-09-15T05:31:42.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:31:42.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-141733920821 2024-09-15T05:31:42.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:31:42.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:31:42.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-141733920821 2024-09-15T05:31:42.413 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:31:42.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=141733920821 2024-09-15T05:31:42.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 141733920821' 2024-09-15T05:31:42.415 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 141733920821 2024-09-15T05:31:42.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:31:42.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 141733920821 -lt 141733920821 2024-09-15T05:31:42.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-09-15T05:31:42.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T05:31:43.100 INFO:tasks.workunit.client.0.smithi138.stderr:dumped all 2024-09-15T05:31:43.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-09-15T05:31:43.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-09-15T05:31:43.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-09-15T05:31:43.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-15T05:31:43.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-15T05:31:43.751 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'pool-rep' does not exist 2024-09-15T05:31:43.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:31:43.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:31:43.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:31:43.763 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:31:43.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:31:43.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:31:43.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:31:43.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:31:43.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:31:43.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:31:43.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:31:43.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:31:43.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:31:43.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:31:43.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:31:43.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:31:43.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:31:43.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:31:43.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:31:43.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:31:43.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:31:43.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:43.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:43.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:31:43.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:31:43.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:31:43.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:31:43.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:31:43.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:31:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:31:43.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:31:43.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:31:43.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:31:43.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:31:43.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:31:43.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:31:43.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:31:43.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:31:43.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:31:43.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:31:43.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:31:43.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:31:43.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:31:43.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:43.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:43.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.400988 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:31:43.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:31:43.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:31:43.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-15T05:31:43.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-15T05:31:44.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:31:44.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:31:44.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:31:44.274 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:31:44.275 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:44.275 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:44.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:31:44.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-15T05:31:44.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:31:44.510 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:31:44.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get fsid 2024-09-15T05:31:44.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:31:44.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:31:44.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:31:44.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:31:44.587 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get mon_host 2024-09-15T05:31:44.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-15T05:31:44.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:44.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:31:44.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:31:44.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:31:44.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-15T05:31:44.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:31:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:31:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:31:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:31:45.497 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:31:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:45.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:31:45.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:31:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:31:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-15T05:31:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:31:45.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-09-15T05:31:45.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:45.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-09-15T05:31:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-09-15T05:31:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-09-15T05:31:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-09-15T05:31:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-15T05:31:45.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-15T05:31:45.842 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-09-15T05:31:45.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-09-15T05:31:45.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:31:45.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:31:45.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:31:45.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb992b7c-2a78-40cf-9e49-0864e30fc05c 2024-09-15T05:31:45.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bb992b7c-2a78-40cf-9e49-0864e30fc05c' 2024-09-15T05:31:45.850 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 bb992b7c-2a78-40cf-9e49-0864e30fc05c 2024-09-15T05:31:45.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:31:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBceZmp9YuMxAAyOSCB1vxEbhdffpsjYGmmQ== 2024-09-15T05:31:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBceZmp9YuMxAAyOSCB1vxEbhdffpsjYGmmQ=="}' 2024-09-15T05:31:45.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb992b7c-2a78-40cf-9e49-0864e30fc05c -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:31:46.159 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:31:46.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:31:46.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBceZmp9YuMxAAyOSCB1vxEbhdffpsjYGmmQ== --osd-uuid bb992b7c-2a78-40cf-9e49-0864e30fc05c 2024-09-15T05:31:46.197 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.196+0000 7fb16d922440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:46.205 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.204+0000 7fb16d922440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:46.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.205+0000 7fb16d922440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:46.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.205+0000 7fb16d922440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:31:46.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.205+0000 7fb16d922440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:31:46.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.205+0000 7fb16d922440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:31:46.237 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:46.236+0000 7fb16d922440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-15T05:31:51.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-15T05:31:51.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:31:51.658 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:31:51.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:31:51.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:31:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:31:52.135 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:31:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:31:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:31:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:31:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:31:52.155 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:52.154+0000 7fd7ff51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:52.157 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:52.156+0000 7fd7ff51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:52.157+0000 7fd7ff51f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:52.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:31:52.482 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:31:52.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:52.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:52.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:52.681+0000 7fd7ff51f440 -1 Falling back to public interface 2024-09-15T05:31:52.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:31:53.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:31:53.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:53.865 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:31:53.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:31:53.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:53.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:54.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:31:55.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:31:55.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:55.223 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:31:55.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:31:55.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:55.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:55.236 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:31:55.236+0000 7fd7ff51f440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:31:55.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:31:56.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:31:56.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:56.612 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:31:56.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:31:56.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:56.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:57.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:31:58.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:31:58.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:58.030 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:31:58.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:31:58.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:58.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:58.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:31:59.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:31:59.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:31:59.422 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:31:59.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:31:59.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:31:59.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:31:59.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:00.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:32:00.790 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:32:00.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:00.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:01.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:02.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:02.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:02.257 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:32:02.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:32:02.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:02.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:02.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:03.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:03.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:03.611 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:32:03.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:32:03.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:03.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:04.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:05.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:05.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:05.005 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:32:05.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:32:05.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:05.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:05.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:06.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:06.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:06.389 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:32:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:32:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:06.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:06.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:07.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:07.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:07.894 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:32:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:32:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:07.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:08.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:09.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:09.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:09.504 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:32:09.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:32:09.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:09.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:10.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:11.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:11.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:11.222 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:32:11.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:32:11.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:11.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:12.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:13.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:13.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:13.277 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:32:13.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:32:13.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:13.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:14.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:15.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:15.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:15.066 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:32:15.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:32:15.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:15.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:15.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:16.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:16.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:16.844 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:32:16.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:32:16.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:16.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:17.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:18.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:18.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:18.393 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:32:18.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:32:18.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:18.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:32:18.745 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/414171,v1:127.0.0.1:6803/414171] [v2:127.0.0.1:6804/414171,v1:127.0.0.1:6805/414171] exists,up bb992b7c-2a78-40cf-9e49-0864e30fc05c 2024-09-15T05:32:18.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:32:18.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:32:18.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:32:18.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:32:18.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:32:18.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=98e50e8e-8681-4e71-be81-b9ff4a371c0e 2024-09-15T05:32:18.753 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 98e50e8e-8681-4e71-be81-b9ff4a371c0e 2024-09-15T05:32:18.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 98e50e8e-8681-4e71-be81-b9ff4a371c0e' 2024-09-15T05:32:18.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:32:18.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiceZmm9NoLRAAqxU3ptnqj8iJbZbuAcDZug== 2024-09-15T05:32:18.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiceZmm9NoLRAAqxU3ptnqj8iJbZbuAcDZug=="}' 2024-09-15T05:32:18.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 98e50e8e-8681-4e71-be81-b9ff4a371c0e -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:32:19.254 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:32:19.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:32:19.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiceZmm9NoLRAAqxU3ptnqj8iJbZbuAcDZug== --osd-uuid 98e50e8e-8681-4e71-be81-b9ff4a371c0e 2024-09-15T05:32:19.288 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.286+0000 7fa07b52d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:19.313 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.311+0000 7fa07b52d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:19.314 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.312+0000 7fa07b52d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:19.315 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.312+0000 7fa07b52d440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:32:19.315 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.312+0000 7fa07b52d440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:32:19.315 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.312+0000 7fa07b52d440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:32:19.416 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:19.412+0000 7fa07b52d440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-15T05:32:25.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-15T05:32:25.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:32:25.121 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:32:25.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:32:25.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:32:25.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:32:25.608 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:32:25.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:32:25.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:32:25.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:32:25.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:32:25.629 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:25.628+0000 7f5be843a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:25.631 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:25.630+0000 7f5be843a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:25.632 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:25.631+0000 7f5be843a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:26.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:32:26.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:32:26.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:32:26.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:32:26.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:32:26.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:26.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:32:26.072 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:32:26.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:26.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:26.151 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:26.151+0000 7f5be843a440 -1 Falling back to public interface 2024-09-15T05:32:26.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:27.425 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:32:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:32:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:27.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:28.646 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:28.646+0000 7f5be843a440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:32:28.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:28.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:28.818 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:32:28.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:32:28.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:28.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:29.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:30.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:30.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:30.436 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:32:30.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:32:30.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:30.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:30.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:31.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:31.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:31.821 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:32:31.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:32:31.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:31.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:32.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:33.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:33.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:33.351 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:32:33.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:32:33.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:33.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:33.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:34.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:34.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:34.887 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:32:34.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:32:34.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:34.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:35.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:36.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:36.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:36.292 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:32:36.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:32:36.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:36.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:36.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:37.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:37.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:37.777 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:32:37.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:32:37.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:37.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:38.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:39.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:39.164 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:32:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:32:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:39.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:40.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:40.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:40.713 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:32:40.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:32:40.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:40.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:41.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:42.070 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:32:42.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:42.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:32:42.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:42.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:42.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:43.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:43.770 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:32:43.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:43.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:32:43.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:43.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:44.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:45.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:45.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:45.806 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:32:45.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:32:45.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:45.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:46.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:47.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:47.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:47.524 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:32:47.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:32:47.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:47.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:48.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:49.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:49.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:49.347 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:32:49.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:32:49.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:49.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:49.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:50.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:50.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:50.827 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:32:50.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:32:50.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:50.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:51.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:52.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:52.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:52.275 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:32:52.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:32:52.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:52.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:52.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:32:53.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:32:53.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:32:53.779 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:32:53.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:32:53.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:32:53.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:32:54.135 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/415925,v1:127.0.0.1:6811/415925] [v2:127.0.0.1:6812/415925,v1:127.0.0.1:6813/415925] exists,up 98e50e8e-8681-4e71-be81-b9ff4a371c0e 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:32:54.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:32:54.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:32:54.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ccde3b22-b8e4-4c56-8afd-47cac8f743d7 2024-09-15T05:32:54.142 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 ccde3b22-b8e4-4c56-8afd-47cac8f743d7 2024-09-15T05:32:54.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ccde3b22-b8e4-4c56-8afd-47cac8f743d7' 2024-09-15T05:32:54.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:32:54.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCGceZmgZX+CBAAC3y1bq3JdAiWEmIL9hI29g== 2024-09-15T05:32:54.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCGceZmgZX+CBAAC3y1bq3JdAiWEmIL9hI29g=="}' 2024-09-15T05:32:54.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ccde3b22-b8e4-4c56-8afd-47cac8f743d7 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:32:54.663 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:32:54.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:32:54.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCGceZmgZX+CBAAC3y1bq3JdAiWEmIL9hI29g== --osd-uuid ccde3b22-b8e4-4c56-8afd-47cac8f743d7 2024-09-15T05:32:54.697 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.696+0000 7feb4660c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:54.698 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.698+0000 7feb4660c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:54.699 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.699+0000 7feb4660c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:32:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.699+0000 7feb4660c440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:32:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.699+0000 7feb4660c440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:32:54.700 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.699+0000 7feb4660c440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:32:54.778 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:32:54.778+0000 7feb4660c440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-15T05:33:00.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-15T05:33:00.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:33:00.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:33:00.921 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:33:00.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:33:01.400 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:33:01.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:33:01.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:33:01.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:33:01.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:33:01.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:33:01.421 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:01.420+0000 7f1f27609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:01.445 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:01.445+0000 7f1f27609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:01.446 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:01.446+0000 7f1f27609440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:33:01.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:01.835 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:33:01.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:33:01.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:01.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:01.978 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:01.978+0000 7f1f27609440 -1 Falling back to public interface 2024-09-15T05:33:02.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:03.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:03.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:03.229 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:33:03.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:33:03.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:03.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:03.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:04.484 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:04.483+0000 7f1f27609440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:33:04.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:04.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:04.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:33:04.585 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:33:04.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:04.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:05.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:06.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:06.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:06.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:33:06.059 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:33:06.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:06.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:06.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:07.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:07.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:07.608 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:33:07.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:33:07.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:07.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:08.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:09.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:09.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:09.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:33:09.108 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:33:09.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:09.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:09.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:10.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:10.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:10.618 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:33:10.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:33:10.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:10.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:10.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:11.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:11.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:11.992 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:33:11.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:33:11.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:11.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:12.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:13.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:13.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:13.526 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:33:13.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:33:13.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:13.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:13.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:14.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:14.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:14.992 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:33:14.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:33:14.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:14.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:15.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:16.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:16.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:16.440 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:33:16.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:33:16.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:16.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:16.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:17.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:17.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:17.810 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:33:17.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:33:17.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:17.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:18.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:19.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:19.615 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:19.616 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:33:19.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:33:19.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:19.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:20.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:21.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:21.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:21.361 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:33:21.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:33:21.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:21.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:22.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:23.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:23.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:23.255 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:33:23.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:33:23.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:23.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:24.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:25.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:25.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:25.084 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:33:25.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:33:25.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:25.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:26.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:27.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:27.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:27.390 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:33:27.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:33:27.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:27.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:28.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:29.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:29.366 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:33:29.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:29.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:33:29.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:29.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:33:29.963 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/417705,v1:127.0.0.1:6819/417705] [v2:127.0.0.1:6820/417705,v1:127.0.0.1:6821/417705] exists,up ccde3b22-b8e4-4c56-8afd-47cac8f743d7 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:33:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:33:29.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:33:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b01469b9-d132-4d1c-95be-15a6d565315b 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b01469b9-d132-4d1c-95be-15a6d565315b' 2024-09-15T05:33:29.971 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 b01469b9-d132-4d1c-95be-15a6d565315b 2024-09-15T05:33:29.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:33:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCpceZm4NJdOhAAYHhAnOMa4j6LWirNtRkSQw== 2024-09-15T05:33:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCpceZm4NJdOhAAYHhAnOMa4j6LWirNtRkSQw=="}' 2024-09-15T05:33:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b01469b9-d132-4d1c-95be-15a6d565315b -i td/osd-rep-recov-eio.sh/3/new.json 2024-09-15T05:33:30.903 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:33:30.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-09-15T05:33:30.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCpceZm4NJdOhAAYHhAnOMa4j6LWirNtRkSQw== --osd-uuid b01469b9-d132-4d1c-95be-15a6d565315b 2024-09-15T05:33:30.938 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:30.937+0000 7f0a79289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:31.005 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.004+0000 7f0a79289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:31.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.005+0000 7f0a79289440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:31.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.005+0000 7f0a79289440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:33:31.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.005+0000 7f0a79289440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:33:31.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.005+0000 7f0a79289440 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-09-15T05:33:31.137 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:31.136+0000 7f0a79289440 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-09-15T05:33:37.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-09-15T05:33:37.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:33:37.017 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:33:37.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:33:37.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:33:37.495 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:33:37.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:33:37.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:33:37.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:33:37.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:33:37.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:33:37.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:37.515+0000 7f1e3441a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:37.517 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:37.517+0000 7f1e3441a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:37.518 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:37.518+0000 7f1e3441a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:37.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:33:37.886 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:33:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:37.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:38.035 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:38.035+0000 7f1e3441a440 -1 Falling back to public interface 2024-09-15T05:33:38.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:39.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:39.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:39.249 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:33:39.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:33:39.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:39.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:39.603 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:40.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:33:40.522+0000 7f1e3441a440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:33:40.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:40.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:40.605 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:33:40.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:33:40.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:40.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:41.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:42.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:42.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:42.098 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:33:42.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:33:42.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:42.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:42.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:43.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:43.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:43.702 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:33:43.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:33:43.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:43.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:44.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:45.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:45.310 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:33:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:33:45.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:45.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:45.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:46.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:46.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:46.694 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:33:46.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:33:46.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:46.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:47.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:48.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:48.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:48.077 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:33:48.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:33:48.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:48.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:48.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:49.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:49.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:49.485 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:33:49.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:33:49.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:49.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:49.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:50.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:50.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:50.918 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:33:50.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:33:50.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:50.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:51.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:52.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:52.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:52.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:33:52.400 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:33:52.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:52.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:52.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:53.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:53.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:53.793 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:33:53.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:33:53.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:53.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:54.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:55.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:55.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:55.739 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:33:55.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:33:55.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:55.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:56.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:57.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:57.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:57.628 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:33:57.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:33:57.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:57.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:33:58.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:33:59.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:33:59.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:33:59.631 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:33:59.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:33:59.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:33:59.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:34:00.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:34:01.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:34:01.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:34:01.978 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:34:01.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:34:01.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:34:01.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:34:02.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:34:03.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:34:03.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:34:03.816 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:34:03.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:34:03.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:34:03.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:34:04.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:34:05.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:34:05.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:34:05.334 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:34:05.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:34:05.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:34:05.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:34:05.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:34:06.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:34:06.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:34:06.974 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:34:06.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:34:06.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:34:06.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:34:07.784 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/419459,v1:127.0.0.1:6827/419459] [v2:127.0.0.1:6828/419459,v1:127.0.0.1:6829/419459] exists,up b01469b9-d132-4d1c-95be-15a6d565315b 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:34:07.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:34:07.787 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:34:07.787 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:34:07.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:34:07.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:34:07.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:34:07.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:34:07.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:34:07.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:34:07.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:34:07.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:07.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:34:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:34:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:08.267 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:34:08.268 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:34:08.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:34:08.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:08.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:34:08.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803803 2024-09-15T05:34:08.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803803 2024-09-15T05:34:08.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803' 2024-09-15T05:34:08.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:08.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:34:08.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574870 2024-09-15T05:34:08.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574870 2024-09-15T05:34:08.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574870' 2024-09-15T05:34:08.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:08.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:34:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345934 2024-09-15T05:34:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345934 2024-09-15T05:34:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574870 2-85899345934' 2024-09-15T05:34:08.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:08.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:34:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803803 1-55834574870 2-85899345934 3-128849018887' 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803803 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:08.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:34:08.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803803 2024-09-15T05:34:08.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:08.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803803 2024-09-15T05:34:08.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803803' 2024-09-15T05:34:08.989 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803803 2024-09-15T05:34:08.989 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:09.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803804 -lt 25769803803 2024-09-15T05:34:09.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:09.336 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574870 2024-09-15T05:34:09.337 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:09.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:34:09.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574870 2024-09-15T05:34:09.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:09.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574870 2024-09-15T05:34:09.339 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574870 2024-09-15T05:34:09.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574870' 2024-09-15T05:34:09.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:34:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574870 -lt 55834574870 2024-09-15T05:34:09.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:09.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345934 2024-09-15T05:34:09.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:09.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:34:09.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345934 2024-09-15T05:34:09.817 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:09.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345934 2024-09-15T05:34:09.818 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345934 2024-09-15T05:34:09.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345934' 2024-09-15T05:34:09.818 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:34:10.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345935 -lt 85899345934 2024-09-15T05:34:10.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:10.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018887 2024-09-15T05:34:10.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:10.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:34:10.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018887 2024-09-15T05:34:10.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:10.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-15T05:34:10.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018887' 2024-09-15T05:34:10.195 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018887 2024-09-15T05:34:10.195 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:10.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018887 -lt 128849018887 2024-09-15T05:34:10.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:34:10.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:10.545 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:11.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T05:34:11.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:11.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:11.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T05:34:11.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:11.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:11.348 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:34:11.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:11.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:11.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:11.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:12.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-09-15T05:34:12.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:12.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:12.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:12.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' 5 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:34:12.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:34:13.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:34:13.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:34:13.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T05:34:13.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:34:13.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:13.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:13.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:13.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:13.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-09-15T05:34:13.739 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:13.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:13.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:14.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-09-15T05:34:14.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 5 2024-09-15T05:34:14.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:34:14.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-09-15T05:34:14.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:34:14.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:34:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:34:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:14.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:14.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:14.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:14.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:14.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:14.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:14.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-09-15T05:34:14.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:14.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:14.643 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 7 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:34:15.112 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:34:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:34:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:34:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T05:34:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:34:15.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:15.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:16.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:34:16.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:16.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:16.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T05:34:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:34:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:34:16.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-15T05:34:16.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-09-15T05:34:16.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-09-15T05:34:17.176 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'pool-rep' already exists 2024-09-15T05:34:17.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:34:18.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-09-15T05:34:18.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:34:18.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:34:18.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:34:18.189 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:34:18.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:34:18.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:34:18.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:34:18.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:34:18.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:34:18.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:34:18.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:34:18.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:34:18.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:34:18.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:18.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:34:18.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:34:18.650 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:18.650 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:34:18.650 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:34:18.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:34:18.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:18.651 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:34:18.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803808 2024-09-15T05:34:18.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803808 2024-09-15T05:34:18.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808' 2024-09-15T05:34:18.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:18.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:34:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574874 2024-09-15T05:34:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574874 2024-09-15T05:34:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574874' 2024-09-15T05:34:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:18.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:34:19.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345939 2024-09-15T05:34:19.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345939 2024-09-15T05:34:19.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574874 2-85899345939' 2024-09-15T05:34:19.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:19.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018891 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018891 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803808 1-55834574874 2-85899345939 3-128849018891' 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803808 2024-09-15T05:34:19.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:19.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:34:19.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803808 2024-09-15T05:34:19.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:19.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803808 2024-09-15T05:34:19.154 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803808 2024-09-15T05:34:19.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803808' 2024-09-15T05:34:19.155 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:19.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803808 -lt 25769803808 2024-09-15T05:34:19.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:19.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574874 2024-09-15T05:34:19.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:19.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:34:19.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574874 2024-09-15T05:34:19.499 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:19.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574874 2024-09-15T05:34:19.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574874' 2024-09-15T05:34:19.500 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574874 2024-09-15T05:34:19.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:34:19.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574874 -lt 55834574874 2024-09-15T05:34:19.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:19.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345939 2024-09-15T05:34:19.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:19.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:34:19.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345939 2024-09-15T05:34:19.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:19.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345939 2024-09-15T05:34:19.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345939' 2024-09-15T05:34:19.853 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345939 2024-09-15T05:34:19.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:34:20.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345939 -lt 85899345939 2024-09-15T05:34:20.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:20.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018891 2024-09-15T05:34:20.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:20.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:34:20.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018891 2024-09-15T05:34:20.201 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:20.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018891 2024-09-15T05:34:20.202 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018891 2024-09-15T05:34:20.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018891' 2024-09-15T05:34:20.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:20.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018889 -lt 128849018891 2024-09-15T05:34:20.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:34:21.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:34:21.563 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:21.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018891 -lt 128849018891 2024-09-15T05:34:21.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:34:21.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:21.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:22.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:22.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:22.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:22.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:34:22.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:22.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:22.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:23.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:34:23.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:34:23.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:34:23.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-09-15T05:34:23.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-09-15T05:34:23.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-09-15T05:34:23.210 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-09-15T05:34:23.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:23.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:23.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:23.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-15T05:34:23.403 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:23.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:23.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:23.752 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:23.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:23.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:23.754 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:23.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:23.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:23.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:23.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-15T05:34:23.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-15T05:34:23.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:23.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:23.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-15T05:34:23.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:23.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:23.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:23.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:23.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:23.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:23.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:23.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:23.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:23.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:23.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:24.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:24.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:24.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:24.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-15T05:34:24.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:24.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-15T05:34:24.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-15T05:34:24.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:24.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:24.442 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:24.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:24.445 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:24.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:24.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:24.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:24.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:24.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:24.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:24.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:24.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:24.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-15T05:34:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:24.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:24.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:24.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:24.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:24.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-15T05:34:24.902 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:25.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:25.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:25.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:25.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:25.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:25.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:25.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:25.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:25.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-15T05:34:25.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-15T05:34:25.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:25.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:25.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-15T05:34:25.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:25.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:25.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:25.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:25.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:25.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:25.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:25.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:25.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:25.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:25.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-15T05:34:25.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:25.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-15T05:34:25.614 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-15T05:34:25.614 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:25.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:25.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:25.980 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:25.981 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:25.981 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:25.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:26.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:26.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:26.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:26.060 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:26.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:26.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:26.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-15T05:34:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-15T05:34:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-15T05:34:26.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:26.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:26.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:26.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-15T05:34:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:26.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:26.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:26.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:26.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-15T05:34:26.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:26.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-15T05:34:26.343 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-15T05:34:26.343 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:26.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:26.706 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:26.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:26.713 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:26.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-15T05:34:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-15T05:34:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:26.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-15T05:34:26.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:26.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:26.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:26.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:26.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:26.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-15T05:34:26.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:26.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:26.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:26.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-15T05:34:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:27.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:27.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:27.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:27.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:27.423 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:27.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:27.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:27.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:27.425 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:27.426 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:27.426 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:27.426 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:27.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:27.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:27.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:27.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:27.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:27.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:27.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:27.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:27.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:27.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:27.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:27.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:27.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:27.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-15T05:34:27.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-15T05:34:27.768 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:28.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:28.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:28.127 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:28.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:28.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:28.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:28.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-15T05:34:28.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:28.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:28.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:28.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:28.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:28.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:28.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:28.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:28.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:28.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-15T05:34:28.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:28.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-15T05:34:28.505 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-15T05:34:28.505 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:28.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:28.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:28.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:28.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:28.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:28.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:28.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:29.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:29.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:29.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:29.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:29.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-15T05:34:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:29.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:29.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:29.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:29.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:29.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:29.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:29.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:29.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-15T05:34:29.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:29.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:29.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:29.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-15T05:34:30.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-15T05:34:30.093 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:30.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:30.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:30.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:30.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:30.441 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:30.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:30.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:30.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:30.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:30.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:30.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:30.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:30.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:30.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:30.523 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:30.524 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:30.524 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:30.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:30.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:30.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:30.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:30.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:30.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:30.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:30.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:34:30.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:30.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:34:30.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:34:30.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:34:30.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:34:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:30.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-15T05:34:30.787 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:31.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:31.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:31.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:31.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:31.133 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:31.136 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:31.136 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:31.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:31.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:31.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:31.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:31.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:31.217 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:31.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:31.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:31.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:31.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:31.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-09-15T05:34:31.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-09-15T05:34:31.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-09-15T05:34:31.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-09-15T05:34:31.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-15T05:34:31.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-09-15T05:34:31.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-09-15T05:34:31.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:31.691 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-15T05:34:31.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-15T05:34:31.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:32.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:34:32.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:34:32.043 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:34:32.043 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:34:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:34:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:34:32.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:34:32.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:34:32.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:34:32.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:34:32.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:34:32.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:32.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:32.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:34:32.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803812 2024-09-15T05:34:32.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803812 2024-09-15T05:34:32.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812' 2024-09-15T05:34:32.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:32.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:34:32.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574878 2024-09-15T05:34:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574878 2024-09-15T05:34:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574878' 2024-09-15T05:34:32.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:32.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:34:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-15T05:34:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-15T05:34:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574878 2-85899345943' 2024-09-15T05:34:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:32.835 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:34:32.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018896 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018896 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803812 1-55834574878 2-85899345943 3-128849018896' 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803812 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:32.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:34:32.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803812 2024-09-15T05:34:32.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:32.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803812 2024-09-15T05:34:32.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-09-15T05:34:32.938 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803812 2024-09-15T05:34:32.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:33.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803812 -lt 25769803812 2024-09-15T05:34:33.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:33.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574878 2024-09-15T05:34:33.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:33.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:34:33.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574878 2024-09-15T05:34:33.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:33.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574878 2024-09-15T05:34:33.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574878' 2024-09-15T05:34:33.313 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574878 2024-09-15T05:34:33.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:34:33.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574878 -lt 55834574878 2024-09-15T05:34:33.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345943 2024-09-15T05:34:33.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:33.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:34:33.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345943 2024-09-15T05:34:33.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:33.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-15T05:34:33.658 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345943 2024-09-15T05:34:33.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-15T05:34:33.659 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:34:34.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-15T05:34:34.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:34.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018896 2024-09-15T05:34:34.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:34.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:34:34.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018896 2024-09-15T05:34:34.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:34.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018896 2024-09-15T05:34:34.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018896' 2024-09-15T05:34:34.011 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018896 2024-09-15T05:34:34.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:34.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018896 -lt 128849018896 2024-09-15T05:34:34.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:34:34.355 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:34.356 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:34.821 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:35.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:34:35.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:35.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:35.156 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:35.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:34:35.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:34:35.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:34:35.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-15T05:34:35.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T05:34:35.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-09-15T05:34:35.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-09-15T05:34:35.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-09-15T05:34:35.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:35.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:36.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:34:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803815 2024-09-15T05:34:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803815 2024-09-15T05:34:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815' 2024-09-15T05:34:36.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:36.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:34:36.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574881 2024-09-15T05:34:36.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574881 2024-09-15T05:34:36.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574881' 2024-09-15T05:34:36.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:36.335 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:34:36.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345946 2024-09-15T05:34:36.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345946 2024-09-15T05:34:36.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574881 2-85899345946' 2024-09-15T05:34:36.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:36.437 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:34:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018898 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018898 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803815 1-55834574881 2-85899345946 3-128849018898' 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803815 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:36.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:34:36.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803815 2024-09-15T05:34:36.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:36.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803815 2024-09-15T05:34:36.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803815' 2024-09-15T05:34:36.543 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803815 2024-09-15T05:34:36.544 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:36.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803813 -lt 25769803815 2024-09-15T05:34:36.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:34:37.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:34:37.890 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:38.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803815 -lt 25769803815 2024-09-15T05:34:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:38.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574881 2024-09-15T05:34:38.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:38.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:34:38.248 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574881 2024-09-15T05:34:38.248 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:38.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574881 2024-09-15T05:34:38.250 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574881 2024-09-15T05:34:38.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-15T05:34:38.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:34:38.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-15T05:34:38.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:38.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345946 2024-09-15T05:34:38.609 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345946 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345946 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345946' 2024-09-15T05:34:38.611 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345946 2024-09-15T05:34:38.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:34:38.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345946 -lt 85899345946 2024-09-15T05:34:38.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:38.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018898 2024-09-15T05:34:38.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:38.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:34:38.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018898 2024-09-15T05:34:38.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:39.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018898 2024-09-15T05:34:39.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018898' 2024-09-15T05:34:39.000 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018898 2024-09-15T05:34:39.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:39.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018899 -lt 128849018898 2024-09-15T05:34:39.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-15T05:34:39.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T05:34:39.725 INFO:tasks.workunit.client.0.smithi138.stderr:dumped all 2024-09-15T05:34:39.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-09-15T05:34:39.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-09-15T05:34:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-09-15T05:34:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-15T05:34:40.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-09-15T05:34:40.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-09-15T05:34:40.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-09-15T05:34:41.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-09-15T05:34:41.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-09-15T05:34:41.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-09-15T05:34:41.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-09-15T05:34:41.755 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-09-15T05:34:41.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:41.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:41.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:41.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:41.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:41.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:41.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:41.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:41.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-15T05:34:41.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:41.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:41.761 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-15T05:34:41.762 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-15T05:34:41.762 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:42.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:42.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:42.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:42.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:42.122 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:42.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:42.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:42.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:42.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:42.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-1 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-1 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:42.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-09-15T05:34:42.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-09-15T05:34:42.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:42.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:42.630 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:42.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:42.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:42.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:42.633 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:42.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:42.635 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:42.636 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:42.636 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:42.636 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:42.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:42.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:42.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:42.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:42.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-09-15T05:34:42.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:42.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:42.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:42.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:42.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:42.803 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:42.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:42.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:42.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-09-15T05:34:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-09-15T05:34:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:42.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:42.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:42.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:42.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-15T05:34:42.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:42.949 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-15T05:34:42.949 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-15T05:34:42.949 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:43.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:43.310 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:43.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:43.313 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:43.314 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:43.314 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:43.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:43.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:43.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:43.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:43.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:43.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:43.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:43.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-2 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-2 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-09-15T05:34:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:43.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:43.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:43.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:43.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:43.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:43.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:43.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:43.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:43.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:43.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:43.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:43.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:43.845 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:43.845 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:43.845 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:43.845 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:43.846 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:43.846 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:43.846 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:43.846 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:43.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:43.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:43.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:43.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-09-15T05:34:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:44.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:44.004 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:44.005 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:44.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:44.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:44.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:44.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-15T05:34:44.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-15T05:34:44.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:44.472 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:44.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:44.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:44.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:44.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-09-15T05:34:44.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:44.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:44.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:44.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-3 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-3 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-09-15T05:34:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-09-15T05:34:44.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:44.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:44.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:44.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:44.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:45.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:45.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:45.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:45.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:45.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-09-15T05:34:45.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:45.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:45.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:45.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:45.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:45.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:45.152 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:45.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:45.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:45.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:45.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:45.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-15T05:34:45.275 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:45.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-15T05:34:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-15T05:34:45.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:45.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:45.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:45.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:45.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:45.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:45.626 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:45.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:45.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:45.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:45.706 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:45.707 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:45.707 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:45.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:45.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-4 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-4 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:45.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-09-15T05:34:45.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-09-15T05:34:45.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:46.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:46.139 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:46.139 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:46.140 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:46.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:46.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:46.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:46.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:46.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:46.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:46.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:46.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:46.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:46.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:46.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-09-15T05:34:46.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:46.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:46.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:46.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:46.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:46.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:46.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:46.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:46.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-15T05:34:46.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:46.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-15T05:34:46.426 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-15T05:34:46.426 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:46.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:46.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:46.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:46.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:46.776 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:46.777 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:46.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:46.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:46.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:46.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:46.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:46.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:46.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:46.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:46.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-09-15T05:34:46.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:46.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-5 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-5 2024-09-15T05:34:46.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:46.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-09-15T05:34:46.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-09-15T05:34:46.939 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:47.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:47.287 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:47.288 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:47.288 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:47.288 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:47.288 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:47.288 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:47.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:47.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:47.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:47.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-09-15T05:34:47.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:47.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:47.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:47.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:47.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:47.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:47.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:47.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:47.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:47.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:47.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:47.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-09-15T05:34:47.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:47.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:47.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:47.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:47.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-15T05:34:47.567 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:47.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:47.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:47.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:47.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:47.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:47.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:47.965 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:48.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:48.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:48.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:48.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:48.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:48.046 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:48.046 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:48.047 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:48.047 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:48.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:48.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:48.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-09-15T05:34:48.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-6 2024-09-15T05:34:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-6 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-09-15T05:34:48.130 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:48.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:48.475 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:48.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:48.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:48.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:48.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:48.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:48.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:48.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-09-15T05:34:48.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:48.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:48.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:48.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:48.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:48.640 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:48.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:48.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:48.640 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:48.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:48.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:48.641 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:48.641 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:48.641 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:48.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:48.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:48.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:48.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:48.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:48.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:48.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:48.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-15T05:34:48.758 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:48.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-15T05:34:48.759 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-15T05:34:48.759 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:49.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:49.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:49.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:49.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:49.111 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:49.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:49.113 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:49.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:49.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:49.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:49.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:49.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-09-15T05:34:49.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:49.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-7 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:49.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:49.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-7 2024-09-15T05:34:49.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:49.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-09-15T05:34:49.279 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-09-15T05:34:49.279 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:49.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:49.630 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:49.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:49.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:49.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:49.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:49.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:49.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:49.633 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:49.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:49.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:49.711 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:49.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:49.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:49.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:49.793 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:49.793 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:49.793 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:49.793 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:49.794 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:49.794 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:49.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:49.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:49.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:49.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-15T05:34:49.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-15T05:34:49.909 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:50.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:50.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:50.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:50.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:50.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:50.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:50.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:50.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-8 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-8 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-09-15T05:34:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:50.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:50.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:50.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:50.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:50.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:50.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:50.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:50.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:50.885 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:50.886 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:50.886 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:50.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:50.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-09-15T05:34:50.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:50.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:50.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:50.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:50.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:50.967 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:50.967 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:50.967 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:50.967 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:50.968 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:50.968 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:50.968 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:50.968 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:50.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:51.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:51.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:51.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:51.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-15T05:34:51.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-15T05:34:51.091 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:51.458 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:51.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:51.461 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:51.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:51.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:51.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:51.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:51.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:51.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-9 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:51.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-9 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-09-15T05:34:51.625 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:51.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:51.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:52.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:52.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:52.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:52.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-09-15T05:34:52.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:52.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:52.135 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:52.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:52.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:52.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-15T05:34:52.260 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:52.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:52.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:52.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:52.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:52.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:52.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:52.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:52.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:52.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-09-15T05:34:52.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-10 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:52.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-10 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-09-15T05:34:52.771 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:53.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:53.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:53.120 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:53.121 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:53.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:53.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:53.202 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:53.203 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:53.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:53.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:53.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-09-15T05:34:53.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:53.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:53.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:53.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:53.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:53.285 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:53.286 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:53.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:53.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:53.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:53.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:53.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-15T05:34:53.399 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:53.796 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=3 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:53.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:53.799 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:53.800 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:53.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:53.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:53.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:53.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.3 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:53.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=pool-rep 2024-09-15T05:34:53.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj-warn-11 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds pool-rep obj-warn-11 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-09-15T05:34:53.963 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-09-15T05:34:53.964 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:34:54.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('3' '1' '0') 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:34:54.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:54.313 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:54.314 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:54.314 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:54.314 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:54.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:54.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:34:54.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:34:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:54.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:34:54.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:34:54.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-09-15T05:34:54.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-09-15T05:34:54.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:34:54.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-09-15T05:34:54.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-09-15T05:34:54.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.3.asok 2024-09-15T05:34:54.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-09-15T05:34:54.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:34:54.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-09-15T05:34:54.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-09-15T05:34:54.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:54.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:54.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:34:54.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:54.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:34:55.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:55.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:34:55.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803821 2024-09-15T05:34:55.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803821 2024-09-15T05:34:55.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821' 2024-09-15T05:34:55.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:55.182 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:34:55.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574887 2024-09-15T05:34:55.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574887 2024-09-15T05:34:55.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-55834574887' 2024-09-15T05:34:55.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:55.299 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:34:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345952 2024-09-15T05:34:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345952 2024-09-15T05:34:55.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-55834574887 2-85899345952' 2024-09-15T05:34:55.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:34:55.402 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:34:55.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018904 2024-09-15T05:34:55.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018904 2024-09-15T05:34:55.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803821 1-55834574887 2-85899345952 3-128849018904' 2024-09-15T05:34:55.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:55.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803821 2024-09-15T05:34:55.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:55.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:34:55.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803821 2024-09-15T05:34:55.505 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:55.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803821 2024-09-15T05:34:55.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803821' 2024-09-15T05:34:55.506 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803821 2024-09-15T05:34:55.506 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:55.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803819 -lt 25769803821 2024-09-15T05:34:55.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:34:56.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:34:56.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:34:57.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803821 -lt 25769803821 2024-09-15T05:34:57.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:57.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574887 2024-09-15T05:34:57.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:57.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:34:57.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574887 2024-09-15T05:34:57.222 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:57.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574887 2024-09-15T05:34:57.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574887' 2024-09-15T05:34:57.224 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574887 2024-09-15T05:34:57.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:34:57.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574887 -lt 55834574887 2024-09-15T05:34:57.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:57.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345952 2024-09-15T05:34:57.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:57.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:34:57.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345952 2024-09-15T05:34:57.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345952 2024-09-15T05:34:57.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345952' 2024-09-15T05:34:57.593 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345952 2024-09-15T05:34:57.593 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:34:57.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345952 -lt 85899345952 2024-09-15T05:34:57.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:34:57.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018904 2024-09-15T05:34:57.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:34:57.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:34:57.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018904 2024-09-15T05:34:57.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:34:57.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018904 2024-09-15T05:34:57.939 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018904 2024-09-15T05:34:57.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018904' 2024-09-15T05:34:57.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:34:58.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018905 -lt 128849018904 2024-09-15T05:34:58.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:34:58.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:58.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:58.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:34:58.753 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:34:59.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:34:59.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:34:59.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:34:59.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:34:59.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:34:59.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:34:59.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:34:59.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-09-15T05:34:59.564 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-09-15T05:34:59.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-09-15T05:34:59.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-15T05:34:59.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-09-15T05:34:59.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-09-15T05:34:59.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:34:59.672 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:3' 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:35:00.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:35:00.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:35:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803824 2024-09-15T05:35:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803824 2024-09-15T05:35:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824' 2024-09-15T05:35:00.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:35:00.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:35:00.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574890 2024-09-15T05:35:00.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574890 2024-09-15T05:35:00.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574890' 2024-09-15T05:35:00.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:35:00.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:35:00.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345955 2024-09-15T05:35:00.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345955 2024-09-15T05:35:00.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-85899345955' 2024-09-15T05:35:00.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:35:00.354 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018907 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018907 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803824 1-55834574890 2-85899345955 3-128849018907' 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803824 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:35:00.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:35:00.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803824 2024-09-15T05:35:00.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:35:00.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803824 2024-09-15T05:35:00.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803824' 2024-09-15T05:35:00.458 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803824 2024-09-15T05:35:00.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:35:00.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803822 -lt 25769803824 2024-09-15T05:35:00.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:35:01.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:35:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:35:02.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803824 -lt 25769803824 2024-09-15T05:35:02.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:35:02.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574890 2024-09-15T05:35:02.162 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:35:02.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:35:02.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574890 2024-09-15T05:35:02.163 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:35:02.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574890 2024-09-15T05:35:02.164 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574890 2024-09-15T05:35:02.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574890' 2024-09-15T05:35:02.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:35:02.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574890 -lt 55834574890 2024-09-15T05:35:02.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:35:02.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345955 2024-09-15T05:35:02.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:35:02.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:35:02.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345955 2024-09-15T05:35:02.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:35:02.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345955 2024-09-15T05:35:02.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345955' 2024-09-15T05:35:02.513 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345955 2024-09-15T05:35:02.513 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:35:02.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345955 -lt 85899345955 2024-09-15T05:35:02.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:35:02.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-128849018907 2024-09-15T05:35:02.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:35:02.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:35:02.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-128849018907 2024-09-15T05:35:02.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:35:02.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018907 2024-09-15T05:35:02.895 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 128849018907 2024-09-15T05:35:02.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 128849018907' 2024-09-15T05:35:02.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:35:03.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018908 -lt 128849018907 2024-09-15T05:35:03.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-09-15T05:35:03.298 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-09-15T05:35:03.574 INFO:tasks.workunit.client.0.smithi138.stderr:dumped all 2024-09-15T05:35:03.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-09-15T05:35:03.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-09-15T05:35:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-09-15T05:35:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-09-15T05:35:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-09-15T05:35:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-09-15T05:35:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-09-15T05:35:04.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-09-15T05:35:04.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-09-15T05:35:05.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-09-15T05:35:05.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-09-15T05:35:05.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-09-15T05:35:05.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-09-15T05:35:05.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-09-15T05:35:05.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-09-15T05:35:06.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-09-15T05:35:06.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-09-15T05:35:06.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-09-15T05:35:06.475 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'pool-rep' does not exist 2024-09-15T05:35:06.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:35:06.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:35:06.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:35:06.489 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:35:06.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:35:06.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:35:06.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:35:06.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:35:06.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:35:06.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:35:06.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:35:06.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:35:06.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:35:06.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:35:06.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:35:06.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:35:06.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:35:06.600 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:35:06.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:35:06.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:35:06.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:35:06.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:06.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:06.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:35:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:35:06.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:35:06.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:35:06.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:35:06.628 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:35:06.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:35:06.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:35:06.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:35:06.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:35:06.630 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:35:06.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:35:06.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:35:06.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:35:06.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:35:06.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:35:06.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:35:06.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:35:06.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:06.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:06.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:35:06.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:35:06.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:35:06.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.636 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:35:06.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:06.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:06.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.400988 2024-09-15T05:35:06.638 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:35:06.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-15T05:35:06.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:07.012 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:07.013 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:07.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-15T05:35:07.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:35:07.264 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:35:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:35:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:07.265 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:07.266 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:35:07.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:35:07.266 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get fsid 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:35:07.342 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:35:07.343 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:35:07.343 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:07.343 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:07.343 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:35:07.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:35:07.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get mon_host 2024-09-15T05:35:07.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-15T05:35:07.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:07.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:35:07.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:35:07.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:35:07.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-15T05:35:07.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:08.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:35:08.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:35:08.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-15T05:35:08.525 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:35:08.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-09-15T05:35:08.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:08.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-09-15T05:35:08.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-15T05:35:08.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-15T05:35:08.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-15T05:35:08.537 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-15T05:35:08.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-15T05:35:08.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:35:08.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:35:08.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:08.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:35:08.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:35:08.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:08.549 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:35:08.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:35:08.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:35:08.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:35:08.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:35:08.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d04eee47-97fb-4719-a96a-1babe9ef554d 2024-09-15T05:35:08.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d04eee47-97fb-4719-a96a-1babe9ef554d' 2024-09-15T05:35:08.553 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 d04eee47-97fb-4719-a96a-1babe9ef554d 2024-09-15T05:35:08.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:35:08.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAMcuZm7eIPIRAAGbZawIHoBIPJ0NoYQG4Y1g== 2024-09-15T05:35:08.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAMcuZm7eIPIRAAGbZawIHoBIPJ0NoYQG4Y1g=="}' 2024-09-15T05:35:08.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d04eee47-97fb-4719-a96a-1babe9ef554d -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:35:08.853 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:35:08.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:35:08.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAMcuZm7eIPIRAAGbZawIHoBIPJ0NoYQG4Y1g== --osd-uuid d04eee47-97fb-4719-a96a-1babe9ef554d 2024-09-15T05:35:08.892 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.889+0000 7f69e671f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:08.899 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.898+0000 7f69e671f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:08.900 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.899+0000 7f69e671f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:08.900 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.899+0000 7f69e671f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:35:08.900 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.899+0000 7f69e671f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:35:08.900 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.899+0000 7f69e671f440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:35:08.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:08.930+0000 7f69e671f440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-15T05:35:14.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-15T05:35:14.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:35:14.230 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:35:14.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:35:14.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:35:14.710 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:35:14.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:35:14.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:35:14.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:35:14.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:35:14.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:35:14.735 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:14.734+0000 7f75e1a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:14.737 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:14.736+0000 7f75e1a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:14.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:14.737+0000 7f75e1a3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:35:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:35:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:35:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:35:15.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:35:15.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:15.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:35:15.108 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:35:15.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:15.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:15.254 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:15.254+0000 7f75e1a3b440 -1 Falling back to public interface 2024-09-15T05:35:15.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:16.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:16.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:16.459 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:35:16.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:35:16.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:16.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:16.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:17.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:17.737+0000 7f75e1a3b440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:35:17.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:17.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:17.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:35:17.824 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:35:17.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:17.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:19.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:19.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:19.202 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:35:19.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:35:19.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:19.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:19.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:20.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:20.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:20.711 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:35:20.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:35:20.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:20.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:21.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:22.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:22.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:22.100 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:35:22.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:35:22.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:22.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:22.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:23.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:23.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:23.558 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:35:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:35:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:23.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:24.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:24.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:24.937 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:35:24.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:35:24.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:24.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:25.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:26.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:26.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:26.401 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:35:26.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:35:26.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:26.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:26.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:27.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:27.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:27.784 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:35:27.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:35:27.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:27.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:28.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:29.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:29.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:29.168 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:35:29.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:35:29.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:29.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:29.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:30.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:30.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:30.976 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:35:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:35:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:31.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:32.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:32.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:32.530 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:35:32.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:35:32.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:32.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:33.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:34.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:34.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:34.260 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:35:34.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:35:34.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:34.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:35.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:36.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:36.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:36.464 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:35:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:35:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:37.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:38.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:38.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:38.153 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:35:38.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:35:38.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:38.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:38.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:39.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:39.679 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:35:39.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:39.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:35:39.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:39.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:40.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:41.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:41.100 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:35:41.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:41.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:35:41.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:41.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:35:41.465 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/426151,v1:127.0.0.1:6803/426151] [v2:127.0.0.1:6804/426151,v1:127.0.0.1:6805/426151] exists,up d04eee47-97fb-4719-a96a-1babe9ef554d 2024-09-15T05:35:41.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:35:41.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:35:41.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:35:41.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:35:41.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:35:41.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bd91ce5d-39a2-4ddd-a29b-342aacc88270 2024-09-15T05:35:41.473 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 bd91ce5d-39a2-4ddd-a29b-342aacc88270 2024-09-15T05:35:41.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 bd91ce5d-39a2-4ddd-a29b-342aacc88270' 2024-09-15T05:35:41.473 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:35:41.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtcuZm+ifYHBAAE6C857KXUbvW9wB1rrI4LQ== 2024-09-15T05:35:41.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtcuZm+ifYHBAAE6C857KXUbvW9wB1rrI4LQ=="}' 2024-09-15T05:35:41.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bd91ce5d-39a2-4ddd-a29b-342aacc88270 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:35:42.068 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:35:42.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:35:42.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtcuZm+ifYHBAAE6C857KXUbvW9wB1rrI4LQ== --osd-uuid bd91ce5d-39a2-4ddd-a29b-342aacc88270 2024-09-15T05:35:42.105 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.104+0000 7fba36921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:42.106 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.106+0000 7fba36921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:42.107 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.106+0000 7fba36921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:42.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.107+0000 7fba36921440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:35:42.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.107+0000 7fba36921440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:35:42.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.107+0000 7fba36921440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:35:42.183 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:42.182+0000 7fba36921440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-15T05:35:47.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-15T05:35:47.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:35:47.971 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:35:47.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:35:47.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:35:48.452 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:35:48.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:35:48.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:35:48.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:35:48.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:35:48.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:35:48.477 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:48.476+0000 7f5fda613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:48.479 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:48.478+0000 7f5fda613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:48.480 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:48.479+0000 7f5fda613440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:35:48.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:35:48.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:35:48.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:48.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:48.995 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:48.994+0000 7f5fda613440 -1 Falling back to public interface 2024-09-15T05:35:49.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:50.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:50.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:50.159 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:35:50.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:35:50.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:50.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:50.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:51.458 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:35:51.457+0000 7f5fda613440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:35:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:51.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:51.544 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:35:51.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:35:51.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:51.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:52.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:53.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:53.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:53.085 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:35:53.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:35:53.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:53.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:53.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:54.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:54.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:54.565 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:35:54.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:35:54.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:54.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:55.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:56.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:56.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:56.055 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:35:56.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:35:56.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:56.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:56.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:57.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:57.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:57.588 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:35:57.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:35:57.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:57.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:58.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:35:59.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:35:59.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:35:59.060 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:35:59.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:35:59.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:35:59.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:35:59.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:00.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:00.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:00.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:36:00.517 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:36:00.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:00.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:01.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:02.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:02.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:02.024 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:36:02.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:36:02.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:02.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:02.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:03.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:03.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:03.609 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:36:03.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:36:03.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:03.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:04.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:05.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:05.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:05.188 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:36:05.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:36:05.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:05.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:06.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:07.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:07.397 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:36:07.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:07.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:36:07.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:07.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:08.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:09.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:09.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:09.096 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:36:09.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:36:09.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:09.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:10.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:11.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:11.094 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:36:11.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:11.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:36:11.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:11.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:11.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:12.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:12.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:12.922 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:36:12.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:36:12.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:12.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:13.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:14.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:14.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:14.639 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:36:14.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:36:14.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:14.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:36:15.178 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/427905,v1:127.0.0.1:6811/427905] [v2:127.0.0.1:6812/427905,v1:127.0.0.1:6813/427905] exists,up bd91ce5d-39a2-4ddd-a29b-342aacc88270 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-15T05:36:15.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:36:15.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:36:15.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=476bdce5-5072-422a-966d-266131d00d38 2024-09-15T05:36:15.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 476bdce5-5072-422a-966d-266131d00d38' 2024-09-15T05:36:15.186 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 476bdce5-5072-422a-966d-266131d00d38 2024-09-15T05:36:15.186 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:36:15.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPcuZmv1KxCxAAv88hd85tnfjs8r77tKQz0g== 2024-09-15T05:36:15.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPcuZmv1KxCxAAv88hd85tnfjs8r77tKQz0g=="}' 2024-09-15T05:36:15.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 476bdce5-5072-422a-966d-266131d00d38 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:36:16.050 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:36:16.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:36:16.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPcuZmv1KxCxAAv88hd85tnfjs8r77tKQz0g== --osd-uuid 476bdce5-5072-422a-966d-266131d00d38 2024-09-15T05:36:16.088 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.086+0000 7f064480d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:16.141 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.139+0000 7f064480d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:16.142 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.140+0000 7f064480d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:16.142 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.140+0000 7f064480d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:36:16.143 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.141+0000 7f064480d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:36:16.143 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.141+0000 7f064480d440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:36:16.189 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:16.188+0000 7f064480d440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-15T05:36:21.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-15T05:36:21.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:36:21.983 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:36:21.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:36:21.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:36:22.467 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:36:22.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:36:22.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:36:22.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:36:22.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:36:22.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:36:22.491 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:22.490+0000 7f04e5c0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:22.513 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:22.512+0000 7f04e5c0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:22.514 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:22.513+0000 7f04e5c0b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:36:22.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:36:22.858 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:36:22.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:22.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:23.048 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:23.047+0000 7f04e5c0b440 -1 Falling back to public interface 2024-09-15T05:36:23.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:24.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:24.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:24.212 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:36:24.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:36:24.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:24.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:24.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:25.538 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:36:25.525+0000 7f04e5c0b440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:36:25.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:25.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:25.585 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:36:25.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:36:25.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:25.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:26.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:27.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:27.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:27.161 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:36:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:36:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:27.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:28.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:28.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:28.668 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:36:28.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:36:28.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:28.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:29.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:30.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:30.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:30.100 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:36:30.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:36:30.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:30.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:30.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:31.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:31.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:31.473 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:36:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:36:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:31.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:32.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:32.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:32.838 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:36:32.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:36:32.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:32.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:33.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:34.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:34.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:34.362 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:36:34.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:36:34.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:34.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:34.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:35.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:35.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:35.825 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:36:35.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:36:35.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:35.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:36.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:37.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:37.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:37.359 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:36:37.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:36:37.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:37.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:37.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:38.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:38.720 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:36:38.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:38.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:36:38.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:38.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:39.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:40.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:40.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:40.270 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:36:40.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:36:40.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:40.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:41.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:42.312 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:36:42.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:42.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:36:42.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:42.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:42.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:43.793 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:36:43.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:36:43.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:43.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:44.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:45.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:45.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:45.430 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:36:45.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:36:45.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:45.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:46.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:47.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:47.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:47.122 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:36:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:36:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:47.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:47.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:36:48.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:36:48.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:36:48.900 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:36:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:36:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:36:48.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:36:49.458 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/429634,v1:127.0.0.1:6819/429634] [v2:127.0.0.1:6820/429634,v1:127.0.0.1:6821/429634] exists,up 476bdce5-5072-422a-966d-266131d00d38 2024-09-15T05:36:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:36:49.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:36:49.461 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:36:49.462 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:36:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:36:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:36:49.462 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:36:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:36:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:36:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:36:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:36:49.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:36:49.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:36:50.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:36:50.030 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:36:50.030 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:36:50.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:36:50.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:50.030 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:36:50.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803798 2024-09-15T05:36:50.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803798 2024-09-15T05:36:50.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803798' 2024-09-15T05:36:50.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:50.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:36:50.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574859 2024-09-15T05:36:50.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574859 2024-09-15T05:36:50.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803798 1-55834574859' 2024-09-15T05:36:50.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:50.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378634 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378634 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803798 1-55834574859 2-81604378634' 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803798 2024-09-15T05:36:50.476 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:36:50.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803798 2024-09-15T05:36:50.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:50.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803798 2024-09-15T05:36:50.479 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803798 2024-09-15T05:36:50.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2024-09-15T05:36:50.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:36:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803796 -lt 25769803798 2024-09-15T05:36:50.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:36:51.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:36:51.883 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:36:52.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803799 -lt 25769803798 2024-09-15T05:36:52.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:52.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574859 2024-09-15T05:36:52.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:52.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:36:52.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574859 2024-09-15T05:36:52.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574859 2024-09-15T05:36:52.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574859' 2024-09-15T05:36:52.246 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574859 2024-09-15T05:36:52.247 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:36:52.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574860 -lt 55834574859 2024-09-15T05:36:52.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:52.596 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378634 2024-09-15T05:36:52.597 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:52.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:36:52.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378634 2024-09-15T05:36:52.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:52.599 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 81604378634 2024-09-15T05:36:52.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378634 2024-09-15T05:36:52.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378634' 2024-09-15T05:36:52.599 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:36:52.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378634 -lt 81604378634 2024-09-15T05:36:52.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:36:52.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:36:52.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:36:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:36:53.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:36:53.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:36:53.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:36:53.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:36:54.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T05:36:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:36:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:36:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-09-15T05:36:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-09-15T05:36:54.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-15T05:36:54.866 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test-pool' already exists 2024-09-15T05:36:54.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:36:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-09-15T05:36:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:36:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:36:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:36:55.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:36:55.879 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:36:55.879 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:36:55.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:36:55.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:36:55.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:36:55.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:36:56.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:36:56.366 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:36:56.366 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:36:56.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:36:56.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:56.367 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:36:56.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803802 2024-09-15T05:36:56.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803802 2024-09-15T05:36:56.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802' 2024-09-15T05:36:56.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:56.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:36:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574863 2024-09-15T05:36:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574863 2024-09-15T05:36:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802 1-55834574863' 2024-09-15T05:36:56.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:36:56.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378637 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378637 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803802 1-55834574863 2-81604378637' 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803802 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:56.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:36:56.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803802 2024-09-15T05:36:56.760 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:56.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803802 2024-09-15T05:36:56.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803802' 2024-09-15T05:36:56.762 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803802 2024-09-15T05:36:56.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:36:57.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803799 -lt 25769803802 2024-09-15T05:36:57.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:36:58.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:36:58.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:36:58.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803802 -lt 25769803802 2024-09-15T05:36:58.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:58.490 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574863 2024-09-15T05:36:58.491 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:58.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:36:58.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574863 2024-09-15T05:36:58.492 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:58.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574863 2024-09-15T05:36:58.493 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574863 2024-09-15T05:36:58.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574863' 2024-09-15T05:36:58.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:36:58.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574863 -lt 55834574863 2024-09-15T05:36:58.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:36:58.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378637 2024-09-15T05:36:58.875 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:36:58.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:36:58.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378637 2024-09-15T05:36:58.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:36:58.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378637 2024-09-15T05:36:58.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378637' 2024-09-15T05:36:58.879 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 81604378637 2024-09-15T05:36:58.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:36:59.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378637 -lt 81604378637 2024-09-15T05:36:59.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:36:59.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:36:59.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:36:59.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:36:59.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:37:00.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:37:00.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:37:00.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:37:00.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:37:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:37:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:37:00.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:37:00.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018263+0000 0'0 22:25 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:31:49.547766+0000 0 0 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018263+0000 0'0 22:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:22:53.803838+0000 0 0 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018298+0000 0'0 22:13 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T17:25:08.019132+0000 0 0 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926082+0000 0'0 22:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:12:33.568694+0000 0 0 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:56.598115+0000 0'0 23:8 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T05:36:54.622810+0000 0'0 2024-09-15T05:36:54.622810+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:21:15.595832+0000 0 0 2024-09-15T05:37:00.879 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926370+0000 0'0 22:25 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:33:50.065618+0000 0 0 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.622402+0000 0'0 22:37 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:29:22.665999+0000 0 0 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.925731+0000 0'0 22:13 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:38:50.732312+0000 0 0 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926045+0000 0'0 22:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:19:03.469156+0000 0 0 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:37:00.880 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:37:00.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:37:00.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:01.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-09-15T05:37:01.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:37:01.028 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T05:37:01.028 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-15T05:37:01.029 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-15T05:37:01.377 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:37:01.378 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:37:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:37:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:37:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:37:01.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:37:01.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-09-15T05:37:02.429 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:37:02.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-09-15T05:37:03.050 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:37:03.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018263+0000 0'0 23:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T09:31:49.547766+0000 0 0 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018263+0000 0'0 23:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:22:53.803838+0000 0 0 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:50.018298+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T17:25:08.019132+0000 0 0 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926082+0000 0'0 23:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:12:33.568694+0000 0 0 2024-09-15T05:37:03.395 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:56.598115+0000 0'0 23:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T05:36:54.622810+0000 0'0 2024-09-15T05:36:54.622810+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:21:15.595832+0000 0 0 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926370+0000 0'0 23:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:33:50.065618+0000 0 0 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-09-15T05:36:50.622402+0000 0'0 23:38 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:29:22.665999+0000 0 0 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.925731+0000 0'0 23:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:38:50.732312+0000 0 0 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:36:49.926045+0000 0'0 23:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:19:03.469156+0000 0 0 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:37:03.396 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:37:03.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-15T05:37:03.407 INFO:tasks.workunit.client.0.smithi138.stderr:4+0 records in 2024-09-15T05:37:03.407 INFO:tasks.workunit.client.0.smithi138.stderr:4+0 records out 2024-09-15T05:37:03.407 INFO:tasks.workunit.client.0.smithi138.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00010523 s, 38.9 MB/s 2024-09-15T05:37:03.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-09-15T05:37:03.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:03.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:03.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:03.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:03.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:03.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:03.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:03.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:03.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:03.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.695 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:04.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:04.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:05.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:05.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:06.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:06.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:07.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:07.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:08.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:08.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:09.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:09.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:10.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:10.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:11.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:11.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:12.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:12.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:13.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:13.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:14.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:14.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:15.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:15.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:16.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:16.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:17.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:17.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:18.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:18.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:19.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:19.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:20.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:20.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:21.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:21.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:22.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:22.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:23.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:23.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:24.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:24.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.829 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:25.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:25.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:26.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:26.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:27.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:27.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:28.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:28.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:29.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:29.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:30.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:30.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:31.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:31.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:32.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:32.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:33.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:33.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:34.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:34.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:35.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:35.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:36.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:36.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:37.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:37.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:38.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:38.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:39.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:39.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:40.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:40.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:41.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:41.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:42.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:37:42.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:37:43.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-09-15T05:37:43.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj250 2024-09-15T05:37:43.125 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:37:43.126 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-15T05:37:43.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-15T05:37:43.126 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:37:43.474 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-15T05:37:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:37:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-15T05:37:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:37:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-15T05:37:43.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.2.asok 2024-09-15T05:37:43.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:37:43.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:37:43.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.2.asok 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:37:43.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.2.asok injectdataerr test-pool obj250 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:37:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj250 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj250 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-09-15T05:37:43.639 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:37:44.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:37:44.015 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T05:37:44.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-15T05:37:44.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-15T05:37:44.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:37:44.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:37:44.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:37:44.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:37:44.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:37:44.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:37:44.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:37:44.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:37:44.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:37:44.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:37:44.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:37:44.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:37:44.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:37:44.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:37:44.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:37:44.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr test-pool obj250 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:37:44.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:37:44.183 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:37:44.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:37:44.185 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:37:44.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:37:44.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-15T05:37:44.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:37:44.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:37:44.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:37:44.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:37:44.206 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:37:44.205+0000 7fda6e529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:37:44.207 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:37:44.207+0000 7fda6e529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:37:44.209 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:37:44.208+0000 7fda6e529440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:37:44.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:37:44.552 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:37:44.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:37:44.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:37:44.723 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:37:44.722+0000 7fda6e529440 -1 Falling back to public interface 2024-09-15T05:37:44.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:37:45.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:37:45.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:37:45.907 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:37:45.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:37:45.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:37:45.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:37:46.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:37:47.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:37:47.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:37:47.268 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:37:47.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:37:47.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:37:47.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:37:47.548 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:37:47.547+0000 7fda6e529440 -1 osd.0 24 log_to_monitors true 2024-09-15T05:37:47.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:37:48.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:37:48.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:37:48.624 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:37:48.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:37:48.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:37:48.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:37:49.231 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 30 up_thru 19 down_at 25 last_clean_interval [6,24) [v2:127.0.0.1:6802/437474,v1:127.0.0.1:6803/437474] [v2:127.0.0.1:6804/437474,v1:127.0.0.1:6805/437474] exists,up d04eee47-97fb-4719-a96a-1babe9ef554d 2024-09-15T05:37:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:37:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:37:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:37:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-09-15T05:37:50.057 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T05:37:50.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-09-15T05:38:05.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-09-15T05:38:05.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:05.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:05.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:05.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:05.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:05.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:05.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:05.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:05.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:05.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T05:38:05.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-15T05:38:05.429 INFO:tasks.workunit.client.0.smithi138.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:05.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-15T05:38:06.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:06.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:06.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:06.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:06.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:06.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:06.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T05:38:06.778 INFO:tasks.workunit.client.0.smithi138.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-15T05:38:06.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-15T05:38:07.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:07.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:07.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:07.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:07.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:07.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:08.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:08.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:08.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:08.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T05:38:08.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-15T05:38:08.140 INFO:tasks.workunit.client.0.smithi138.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:08.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-15T05:38:09.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:09.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:09.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:09.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:09.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:09.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:09.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:09.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:09.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:09.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T05:38:09.494 INFO:tasks.workunit.client.0.smithi138.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:09.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-15T05:38:09.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-15T05:38:10.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:10.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:10.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:10.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:10.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:10.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:10.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:10.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:10.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:10.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-09-15T05:38:10.852 INFO:tasks.workunit.client.0.smithi138.stdout:active+undersized+degraded+remapped+backfilling 2024-09-15T05:38:10.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-09-15T05:38:10.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-09-15T05:38:11.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-09-15T05:38:11.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-09-15T05:38:11.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:38:11.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:38:11.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:38:11.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:38:12.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-09-15T05:38:12.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-09-15T05:38:12.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-09-15T05:38:12.215 INFO:tasks.workunit.client.0.smithi138.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-09-15T05:38:12.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-09-15T05:38:12.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-09-15T05:38:12.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.247162+0000 0'0 36:66 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:07:00.603459+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.885597+0000 0'0 37:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:51:47.258217+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.884842+0000 0'0 37:55 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T05:56:49.615859+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.884891+0000 0'0 37:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:59:33.961300+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-09-15T05:38:06.268395+0000 28'301 37:900 [2,1,0] 2 [2,1] 2 0'0 2024-09-15T05:36:54.622810+0000 0'0 2024-09-15T05:36:54.622810+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:28:19.465754+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.247129+0000 0'0 36:66 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:46:21.934607+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.754551+0000 0'0 37:56 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T10:22:31.177900+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.884479+0000 0'0 37:55 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:57:13.455811+0000 0 0 2024-09-15T05:38:12.549 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:37:51.817698+0000 0'0 37:68 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:35:08.411792+0000 0'0 2024-09-15T05:35:08.411792+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:38:31.350313+0000 0 0 2024-09-15T05:38:12.550 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:38:12.550 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:38:12.550 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:38:12.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-09-15T05:38:12.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-09-15T05:38:12.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:17.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-09-15T05:38:17.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T05:38:17.885 INFO:tasks.workunit.client.0.smithi138.stderr:pg has no unfound objects 2024-09-15T05:38:17.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-09-15T05:38:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:38:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:38:17.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:38:17.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:38:17.896 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:38:17.896 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:38:17.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:38:17.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:38:17.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:38:17.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:38:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:38:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:38:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:38:17.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:38:17.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:18.338 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:38:18.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018888 2024-09-15T05:38:18.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018888 2024-09-15T05:38:18.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888' 2024-09-15T05:38:18.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:18.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:38:18.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574881 2024-09-15T05:38:18.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574881 2024-09-15T05:38:18.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888 1-55834574881' 2024-09-15T05:38:18.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:18.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:38:18.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=81604378655 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 81604378655 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-128849018888 1-55834574881 2-81604378655' 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-128849018888 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:18.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:38:18.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-128849018888 2024-09-15T05:38:18.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:18.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018888 2024-09-15T05:38:18.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 128849018888' 2024-09-15T05:38:18.839 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 128849018888 2024-09-15T05:38:18.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:38:19.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018886 -lt 128849018888 2024-09-15T05:38:19.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:38:20.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:38:20.194 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:38:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018889 -lt 128849018888 2024-09-15T05:38:20.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:20.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574881 2024-09-15T05:38:20.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:20.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:38:20.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574881 2024-09-15T05:38:20.704 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:20.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574881 2024-09-15T05:38:20.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-09-15T05:38:20.705 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574881 2024-09-15T05:38:20.706 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:38:21.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574881 -lt 55834574881 2024-09-15T05:38:21.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:21.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-81604378655 2024-09-15T05:38:21.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:21.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:38:21.069 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-81604378655 2024-09-15T05:38:21.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:21.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=81604378655 2024-09-15T05:38:21.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 81604378655' 2024-09-15T05:38:21.071 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 81604378655 2024-09-15T05:38:21.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:38:21.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 81604378655 -lt 81604378655 2024-09-15T05:38:21.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:38:21.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:21.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:22.008 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:22.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:38:22.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:22.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:22.352 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:22.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:38:22.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-09-15T05:38:22.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:38:22.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-09-15T05:38:22.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:38:22.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:22.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:22.949 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:23.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:38:23.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:23.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:23.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:23.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:38:23.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:38:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12800 2024-09-15T05:38:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12800 '!=' null 2024-09-15T05:38:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:38:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:38:24.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:38:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:38:24.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:24.351 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:24.693 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:38:24.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:24.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:24.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:38:25.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:38:25.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12800 2024-09-15T05:38:25.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12800 '!=' null 2024-09-15T05:38:25.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:38:25.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:38:25.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:25.738 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:26.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:38:26.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:26.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:26.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:26.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:38:26.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:38:26.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:38:26.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-09-15T05:38:26.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-09-15T05:38:26.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-09-15T05:38:26.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-09-15T05:38:26.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-09-15T05:38:26.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-09-15T05:38:26.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-09-15T05:38:26.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-09-15T05:38:26.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-09-15T05:38:26.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-09-15T05:38:26.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-09-15T05:38:26.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-09-15T05:38:26.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-09-15T05:38:26.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:26.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:26.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-09-15T05:38:26.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-09-15T05:38:27.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-09-15T05:38:27.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-09-15T05:38:27.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-09-15T05:38:27.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-09-15T05:38:27.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-09-15T05:38:27.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-09-15T05:38:27.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-09-15T05:38:27.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-09-15T05:38:27.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-09-15T05:38:27.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-09-15T05:38:27.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-09-15T05:38:27.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-09-15T05:38:27.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-09-15T05:38:27.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-09-15T05:38:27.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-09-15T05:38:27.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-09-15T05:38:27.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-09-15T05:38:27.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-09-15T05:38:27.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-09-15T05:38:27.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-09-15T05:38:27.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-09-15T05:38:27.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-09-15T05:38:27.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-09-15T05:38:27.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-09-15T05:38:27.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-09-15T05:38:27.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-09-15T05:38:27.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-09-15T05:38:27.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-09-15T05:38:27.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-09-15T05:38:27.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-09-15T05:38:27.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-09-15T05:38:27.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-09-15T05:38:27.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-09-15T05:38:27.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-09-15T05:38:27.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:27.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:27.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-09-15T05:38:27.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-09-15T05:38:28.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-09-15T05:38:28.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-09-15T05:38:28.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-09-15T05:38:28.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-09-15T05:38:28.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-09-15T05:38:28.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-09-15T05:38:28.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-09-15T05:38:28.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-09-15T05:38:28.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-09-15T05:38:28.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-09-15T05:38:28.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-09-15T05:38:28.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-09-15T05:38:28.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-09-15T05:38:28.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-09-15T05:38:28.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-09-15T05:38:28.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-09-15T05:38:28.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-09-15T05:38:28.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-09-15T05:38:28.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-09-15T05:38:28.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-09-15T05:38:28.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-09-15T05:38:28.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-09-15T05:38:28.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-09-15T05:38:28.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-09-15T05:38:28.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-09-15T05:38:28.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-09-15T05:38:28.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-09-15T05:38:28.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-09-15T05:38:28.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-09-15T05:38:28.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:28.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:28.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-09-15T05:38:28.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-09-15T05:38:29.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-09-15T05:38:29.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-09-15T05:38:29.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-09-15T05:38:29.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-09-15T05:38:29.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-09-15T05:38:29.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-09-15T05:38:29.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-09-15T05:38:29.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-09-15T05:38:29.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-09-15T05:38:29.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-09-15T05:38:29.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-09-15T05:38:29.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-09-15T05:38:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-09-15T05:38:29.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-09-15T05:38:29.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-09-15T05:38:29.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-09-15T05:38:29.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-09-15T05:38:29.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-09-15T05:38:29.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-09-15T05:38:29.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-09-15T05:38:29.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-09-15T05:38:29.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-09-15T05:38:29.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-09-15T05:38:29.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-09-15T05:38:29.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-09-15T05:38:29.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-09-15T05:38:29.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-09-15T05:38:29.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-09-15T05:38:29.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-09-15T05:38:29.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-09-15T05:38:29.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-09-15T05:38:29.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-09-15T05:38:29.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-09-15T05:38:29.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-09-15T05:38:29.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-09-15T05:38:29.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-09-15T05:38:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-09-15T05:38:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-09-15T05:38:30.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-09-15T05:38:30.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-09-15T05:38:30.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-09-15T05:38:30.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-09-15T05:38:30.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-09-15T05:38:30.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-09-15T05:38:30.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-09-15T05:38:30.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-09-15T05:38:30.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-09-15T05:38:30.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-09-15T05:38:30.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-09-15T05:38:30.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-09-15T05:38:30.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-09-15T05:38:30.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-09-15T05:38:30.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-09-15T05:38:30.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-09-15T05:38:30.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-09-15T05:38:30.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.601 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-09-15T05:38:30.602 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-09-15T05:38:30.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-09-15T05:38:30.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.682 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-09-15T05:38:30.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-09-15T05:38:30.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-09-15T05:38:30.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-09-15T05:38:30.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-09-15T05:38:30.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-09-15T05:38:30.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-09-15T05:38:30.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-09-15T05:38:30.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-09-15T05:38:30.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-09-15T05:38:30.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-09-15T05:38:30.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:30.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:30.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-09-15T05:38:30.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-09-15T05:38:31.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-09-15T05:38:31.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-09-15T05:38:31.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-09-15T05:38:31.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-09-15T05:38:31.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-09-15T05:38:31.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-09-15T05:38:31.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-09-15T05:38:31.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-09-15T05:38:31.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-09-15T05:38:31.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-09-15T05:38:31.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-09-15T05:38:31.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-09-15T05:38:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-09-15T05:38:31.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-09-15T05:38:31.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-09-15T05:38:31.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-09-15T05:38:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-09-15T05:38:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-09-15T05:38:31.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-09-15T05:38:31.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-09-15T05:38:31.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-09-15T05:38:31.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-09-15T05:38:31.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-09-15T05:38:31.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-09-15T05:38:31.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-09-15T05:38:31.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-09-15T05:38:31.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-09-15T05:38:31.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-09-15T05:38:31.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-09-15T05:38:31.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-09-15T05:38:31.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-09-15T05:38:31.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-09-15T05:38:31.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-09-15T05:38:31.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-09-15T05:38:31.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:31.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:31.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-09-15T05:38:31.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-09-15T05:38:32.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-09-15T05:38:32.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-09-15T05:38:32.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-09-15T05:38:32.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-09-15T05:38:32.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-09-15T05:38:32.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-09-15T05:38:32.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-09-15T05:38:32.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-09-15T05:38:32.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-09-15T05:38:32.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-09-15T05:38:32.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-09-15T05:38:32.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-09-15T05:38:32.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-09-15T05:38:32.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-09-15T05:38:32.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-09-15T05:38:32.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-09-15T05:38:32.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-09-15T05:38:32.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-09-15T05:38:32.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-09-15T05:38:32.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-09-15T05:38:32.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-09-15T05:38:32.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-09-15T05:38:32.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-09-15T05:38:32.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-09-15T05:38:32.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-09-15T05:38:32.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-09-15T05:38:32.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-09-15T05:38:32.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-09-15T05:38:32.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-09-15T05:38:32.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-09-15T05:38:32.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-09-15T05:38:32.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-09-15T05:38:32.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-09-15T05:38:32.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-09-15T05:38:32.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:32.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:32.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-09-15T05:38:32.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-09-15T05:38:33.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-09-15T05:38:33.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-09-15T05:38:33.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-09-15T05:38:33.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-09-15T05:38:33.120 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-09-15T05:38:33.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-09-15T05:38:33.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-09-15T05:38:33.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-09-15T05:38:33.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-09-15T05:38:33.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-09-15T05:38:33.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-09-15T05:38:33.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-09-15T05:38:33.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-09-15T05:38:33.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-09-15T05:38:33.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-09-15T05:38:33.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-09-15T05:38:33.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-09-15T05:38:33.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-09-15T05:38:33.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-09-15T05:38:33.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-09-15T05:38:33.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-09-15T05:38:33.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-09-15T05:38:33.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-09-15T05:38:33.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-09-15T05:38:33.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-09-15T05:38:33.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-09-15T05:38:33.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.741 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-09-15T05:38:33.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-09-15T05:38:33.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-09-15T05:38:33.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-09-15T05:38:33.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-09-15T05:38:33.824 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-09-15T05:38:33.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-09-15T05:38:33.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-09-15T05:38:33.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-09-15T05:38:33.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:33.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:33.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-09-15T05:38:33.973 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-09-15T05:38:34.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-09-15T05:38:34.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-09-15T05:38:34.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-09-15T05:38:34.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-09-15T05:38:34.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-09-15T05:38:34.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-09-15T05:38:34.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-09-15T05:38:34.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-09-15T05:38:34.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-09-15T05:38:34.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-09-15T05:38:34.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-09-15T05:38:34.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-09-15T05:38:34.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-09-15T05:38:34.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-09-15T05:38:34.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-09-15T05:38:34.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-09-15T05:38:34.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-09-15T05:38:34.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-09-15T05:38:34.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-09-15T05:38:34.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-09-15T05:38:34.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-09-15T05:38:34.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-09-15T05:38:34.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-09-15T05:38:34.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-09-15T05:38:34.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-09-15T05:38:34.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-09-15T05:38:34.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-09-15T05:38:34.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-09-15T05:38:34.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-09-15T05:38:34.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-09-15T05:38:34.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-09-15T05:38:34.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-09-15T05:38:34.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-09-15T05:38:34.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-09-15T05:38:34.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:34.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:34.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-09-15T05:38:34.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:35.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-09-15T05:38:35.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:35.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-09-15T05:38:35.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:35.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-09-15T05:38:35.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:35.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-09-15T05:38:35.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:38:35.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-09-15T05:38:35.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:38:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-09-15T05:38:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-15T05:38:35.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-15T05:38:35.601 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test-pool' does not exist 2024-09-15T05:38:35.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:38:35.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:38:35.612 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:38:35.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:38:35.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:38:35.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:38:35.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:38:35.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:38:35.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:38:35.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:38:35.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:38:35.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:38:35.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:38:35.722 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:38:35.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:38:35.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:38:35.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:38:35.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:38:35.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:38:35.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:38:35.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:38:35.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:35.762 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:35.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:38:35.764 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:38:35.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:38:35.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:38:35.766 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:38:35.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:38:35.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:38:35.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:38:35.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:38:35.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:38:35.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:38:35.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:38:35.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:38:35.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:38:35.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.771 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:38:35.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:35.772 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:35.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:38:35.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:38:35.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:38:35.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:38:35.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:35.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:35.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.400988 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-15T05:38:35.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-15T05:38:36.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:36.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-15T05:38:36.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:38:36.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:38:36.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get fsid 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:38:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:38:36.467 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get mon_host 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-15T05:38:36.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:37.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:38:37.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:38:37.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-15T05:38:37.682 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_read_unfound td/osd-rep-recov-eio.sh 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:420: TEST_rep_read_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:421: TEST_rep_read_unfound: local objname=myobject 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:423: TEST_rep_read_unfound: setup_osds 3 _filestore 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type=_filestore 2024-09-15T05:38:37.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-15T05:38:37.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-15T05:38:37.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:38:37.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 0 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=0 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T05:38:37.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T05:38:37.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=de7eeebc-96c3-487a-b3cf-0bbbac463207 2024-09-15T05:38:37.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 de7eeebc-96c3-487a-b3cf-0bbbac463207' 2024-09-15T05:38:37.701 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 de7eeebc-96c3-487a-b3cf-0bbbac463207 2024-09-15T05:38:37.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T05:38:37.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDdcuZmHll3KhAAAL4I8+luwusKtgGad2m/sQ== 2024-09-15T05:38:37.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDdcuZmHll3KhAAAL4I8+luwusKtgGad2m/sQ=="}' 2024-09-15T05:38:37.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new de7eeebc-96c3-487a-b3cf-0bbbac463207 -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:38:38.003 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:38:38.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:38:38.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdcuZmHll3KhAAAL4I8+luwusKtgGad2m/sQ== --osd-uuid de7eeebc-96c3-487a-b3cf-0bbbac463207 --osd-objectstore=filestore 2024-09-15T05:38:38.039 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.037+0000 7f632840b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:38.048 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.046+0000 7f632840b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:38.049 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.048+0000 7f632840b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:38.491 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.489+0000 7f632840b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:38.928 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.926+0000 7f632840b440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:38.928 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.926+0000 7f632840b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:38.928 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:38.926+0000 7f632840b440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:39.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-15T05:38:39.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T05:38:39.508 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:38:39.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd0 key to auth repository 2024-09-15T05:38:39.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:38:39.652 INFO:tasks.workunit.client.0.smithi138.stderr:Error EINVAL: key for osd.0 not found in provided keyring 2024-09-15T05:38:39.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.0 2024-09-15T05:38:39.655 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:38:39.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:39.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T05:38:39.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T05:38:39.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T05:38:39.679 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:39.677+0000 7fea4191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:39.681 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:39.679+0000 7fea4191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:39.682 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:39.680+0000 7fea4191d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:39.683 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:39.682+0000 7fea4191d440 -1 Falling back to public interface 2024-09-15T05:38:39.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 0 2024-09-15T05:38:39.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:38:39.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:38:39.984 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:39.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:38:40.235 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:40.233+0000 7fea4191d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:40.242 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:40.240+0000 7fea4191d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:40.242 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:40.240+0000 7fea4191d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:40.247 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:40.245+0000 7fea4191d440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:38:40.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:38:41.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:38:41.297 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:38:41.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:41.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:38:41.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:41.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:38:41.494 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/445532,v1:127.0.0.1:6801/445532] [v2:127.0.0.1:6802/445532,v1:127.0.0.1:6803/445532] exists,up de7eeebc-96c3-487a-b3cf-0bbbac463207 2024-09-15T05:38:41.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 1 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=1 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:38:41.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:38:41.500 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:38:41.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=68424bb2-38bb-4cdd-8288-8b32bce58246 2024-09-15T05:38:41.502 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 68424bb2-38bb-4cdd-8288-8b32bce58246 2024-09-15T05:38:41.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 68424bb2-38bb-4cdd-8288-8b32bce58246' 2024-09-15T05:38:41.503 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T05:38:41.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDhcuZmOkiXHhAAScwJpc7FC3sbr6hQVkPe9Q== 2024-09-15T05:38:41.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDhcuZmOkiXHhAAScwJpc7FC3sbr6hQVkPe9Q=="}' 2024-09-15T05:38:41.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 68424bb2-38bb-4cdd-8288-8b32bce58246 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:38:42.313 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:38:42.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:38:42.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDhcuZmOkiXHhAAScwJpc7FC3sbr6hQVkPe9Q== --osd-uuid 68424bb2-38bb-4cdd-8288-8b32bce58246 --osd-objectstore=filestore 2024-09-15T05:38:42.351 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:42.349+0000 7f7263929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:42.432 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:42.430+0000 7f7263929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:42.433 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:42.431+0000 7f7263929440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:43.136 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:43.134+0000 7f7263929440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:43.882 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:43.881+0000 7f7263929440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:43.882 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:43.881+0000 7f7263929440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:43.882 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:43.881+0000 7f7263929440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:44.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-15T05:38:44.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T05:38:44.422 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:38:44.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd1 key to auth repository 2024-09-15T05:38:44.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:38:44.737 INFO:tasks.workunit.client.0.smithi138.stderr:Error EINVAL: key for osd.1 not found in provided keyring 2024-09-15T05:38:44.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.1 2024-09-15T05:38:44.741 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:38:44.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:44.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T05:38:44.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T05:38:44.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T05:38:44.764 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:44.763+0000 7f463771d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:44.766 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:44.766+0000 7f463771d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:44.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:44.767+0000 7f463771d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:44.769 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:44.768+0000 7f463771d440 -1 Falling back to public interface 2024-09-15T05:38:45.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 1 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:45.141 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:38:45.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:38:45.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:45.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:38:45.196 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:45.195+0000 7f463771d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:45.204 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:45.203+0000 7f463771d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:45.204 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:45.203+0000 7f463771d440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:45.208 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:45.207+0000 7f463771d440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:38:45.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:38:46.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:38:46.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:46.753 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:38:46.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:38:46.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:46.754 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:38:47.153 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/445894,v1:127.0.0.1:6811/445894] [v2:127.0.0.1:6812/445894,v1:127.0.0.1:6813/445894] exists,up 68424bb2-38bb-4cdd-8288-8b32bce58246 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd_filestore td/osd-rep-recov-eio.sh 2 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:689: run_osd_filestore: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:690: run_osd_filestore: shift 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:691: run_osd_filestore: local id=2 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:692: run_osd_filestore: shift 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:693: run_osd_filestore: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:695: run_osd_filestore: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:38:47.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:696: run_osd_filestore: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:697: run_osd_filestore: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:698: run_osd_filestore: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:699: run_osd_filestore: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:700: run_osd_filestore: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:701: run_osd_filestore: ceph_args+=' --chdir=' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:702: run_osd_filestore: ceph_args+= 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:703: run_osd_filestore: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: get_asok_path 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:38:47.158 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:704: run_osd_filestore: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:705: run_osd_filestore: ceph_args+=' --debug-osd=20' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:706: run_osd_filestore: ceph_args+=' --debug-ms=1' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:707: run_osd_filestore: ceph_args+=' --debug-monc=20' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:708: run_osd_filestore: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:709: run_osd_filestore: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:710: run_osd_filestore: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:711: run_osd_filestore: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:712: run_osd_filestore: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:713: run_osd_filestore: ceph_args+=' ' 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:714: run_osd_filestore: ceph_args+= 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:715: run_osd_filestore: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: uuidgen 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:717: run_osd_filestore: local uuid=13578e50-8db1-4087-b08c-c99925d41891 2024-09-15T05:38:47.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:718: run_osd_filestore: echo 'add osd2 13578e50-8db1-4087-b08c-c99925d41891' 2024-09-15T05:38:47.160 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 13578e50-8db1-4087-b08c-c99925d41891 2024-09-15T05:38:47.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: ceph-authtool --gen-print-key 2024-09-15T05:38:47.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:719: run_osd_filestore: OSD_SECRET=AQDncuZmmBw8ChAATAiTU4gfEOFYuo3HGfj3JA== 2024-09-15T05:38:47.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:720: run_osd_filestore: echo '{"cephx_secret": "AQDncuZmmBw8ChAATAiTU4gfEOFYuo3HGfj3JA=="}' 2024-09-15T05:38:47.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:721: run_osd_filestore: ceph osd new 13578e50-8db1-4087-b08c-c99925d41891 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:38:47.704 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:38:47.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:722: run_osd_filestore: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:38:47.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:723: run_osd_filestore: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDncuZmmBw8ChAATAiTU4gfEOFYuo3HGfj3JA== --osd-uuid 13578e50-8db1-4087-b08c-c99925d41891 --osd-objectstore=filestore 2024-09-15T05:38:47.743 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:47.742+0000 7f133fe0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:47.753 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:47.753+0000 7f133fe0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:47.755 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:47.754+0000 7f133fe0d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:48.417 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:48.416+0000 7f133fe0d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:48.862 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:48.861+0000 7f133fe0d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:48.862 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:48.861+0000 7f133fe0d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:48.862 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:48.861+0000 7f133fe0d440 -1 journal do_read_entry(4096): bad header magic 2024-09-15T05:38:49.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:725: run_osd_filestore: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-15T05:38:49.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:726: run_osd_filestore: cat 2024-09-15T05:38:49.519 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:38:49.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:730: run_osd_filestore: echo adding osd2 key to auth repository 2024-09-15T05:38:49.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:731: run_osd_filestore: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:38:50.010 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:38:50.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:732: run_osd_filestore: echo start osd.2 2024-09-15T05:38:50.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:733: run_osd_filestore: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:38:50.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: ceph osd dump --format=json 2024-09-15T05:38:50.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: jq '.flags_set[]' 2024-09-15T05:38:50.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:736: run_osd_filestore: grep -q '"noup"' 2024-09-15T05:38:50.034 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.033+0000 7f0dd3c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:50.036 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.035+0000 7f0dd3c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:50.037 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.036+0000 7f0dd3c16440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:38:50.038 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.037+0000 7f0dd3c16440 -1 Falling back to public interface 2024-09-15T05:38:50.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:739: run_osd_filestore: wait_for_osd up 2 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:38:50.431 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:38:50.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:50.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:38:50.769 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.768+0000 7f0dd3c16440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:38:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.779+0000 7f0dd3c16440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:50.779 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.779+0000 7f0dd3c16440 -1 journal do_read_entry(8192): bad header magic 2024-09-15T05:38:50.783 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:38:50.783+0000 7f0dd3c16440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:38:50.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:38:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:38:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:38:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:38:51.977 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:38:51.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:38:51.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:38:52.444 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/446178,v1:127.0.0.1:6819/446178] [v2:127.0.0.1:6820/446178,v1:127.0.0.1:6821/446178] exists,up 13578e50-8db1-4087-b08c-c99925d41891 2024-09-15T05:38:52.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:38:52.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:38:52.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:38:52.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:38:52.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:38:52.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:38:52.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:38:52.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:38:52.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:38:52.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:38:52.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:53.081 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:38:53.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803781 2024-09-15T05:38:53.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803781 2024-09-15T05:38:53.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781' 2024-09-15T05:38:53.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:53.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:38:53.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574853 2024-09-15T05:38:53.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574853 2024-09-15T05:38:53.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781 1-55834574853' 2024-09-15T05:38:53.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:38:53.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:38:54.300 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345923 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345923 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803781 1-55834574853 2-85899345923' 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803781 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:54.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:38:54.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803781 2024-09-15T05:38:54.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:54.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803781 2024-09-15T05:38:54.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803781' 2024-09-15T05:38:54.304 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803781 2024-09-15T05:38:54.304 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:38:54.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803779 -lt 25769803781 2024-09-15T05:38:54.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:38:55.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:38:55.668 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:38:56.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803781 -lt 25769803781 2024-09-15T05:38:56.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:56.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574853 2024-09-15T05:38:56.060 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:38:56.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574853 2024-09-15T05:38:56.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574853 2024-09-15T05:38:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574853' 2024-09-15T05:38:56.062 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574853 2024-09-15T05:38:56.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:38:56.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574853 -lt 55834574853 2024-09-15T05:38:56.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:38:56.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345923 2024-09-15T05:38:56.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:38:56.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:38:56.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345923 2024-09-15T05:38:56.417 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:38:56.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345923 2024-09-15T05:38:56.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345923' 2024-09-15T05:38:56.418 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345923 2024-09-15T05:38:56.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:38:56.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345923 -lt 85899345923 2024-09-15T05:38:56.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:38:56.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:56.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:57.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:57.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:57.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-15T05:38:57.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:57.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:57.667 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' -1 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=6 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:38:58.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:38:58.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:38:58.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:58.238 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:38:58.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-09-15T05:38:58.580 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:38:58.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:38:58.581 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:38:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-09-15T05:38:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' 6 2024-09-15T05:38:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:38:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:38:59.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:38:59.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:38:59.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:38:59.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:38:59.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:38:59.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:38:59.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:38:59.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T05:38:59.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:38:59.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T05:38:59.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:38:59.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:38:59.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:38:59.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:38:59.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:38:59.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:38:59.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:38:59.726 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:00.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:39:00.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:00.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:00.070 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T05:39:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:39:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:39:00.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:425: TEST_rep_read_unfound: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-09-15T05:39:01.009 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' does not exist 2024-09-15T05:39:01.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:426: TEST_rep_read_unfound: local poolname=test-pool 2024-09-15T05:39:01.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:427: TEST_rep_read_unfound: create_pool test-pool 1 1 2024-09-15T05:39:01.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-15T05:39:01.631 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test-pool' already exists 2024-09-15T05:39:01.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:39:02.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:428: TEST_rep_read_unfound: ceph osd pool set test-pool size 2 2024-09-15T05:39:03.499 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 2 size to 2 2024-09-15T05:39:03.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:429: TEST_rep_read_unfound: wait_for_clean 2024-09-15T05:39:03.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:39:03.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:39:03.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:39:03.510 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:39:03.511 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:39:03.511 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:39:03.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:39:03.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:39:03.511 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:39:03.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:03.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:39:04.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803785 2024-09-15T05:39:04.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803785 2024-09-15T05:39:04.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803785' 2024-09-15T05:39:04.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:04.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:39:04.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574857 2024-09-15T05:39:04.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574857 2024-09-15T05:39:04.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803785 1-55834574857' 2024-09-15T05:39:04.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:04.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:39:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345927 2024-09-15T05:39:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345927 2024-09-15T05:39:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803785 1-55834574857 2-85899345927' 2024-09-15T05:39:04.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:04.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803785 2024-09-15T05:39:04.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:04.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:39:04.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803785 2024-09-15T05:39:04.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:04.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803785 2024-09-15T05:39:04.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803785' 2024-09-15T05:39:04.274 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803785 2024-09-15T05:39:04.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:04.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803783 -lt 25769803785 2024-09-15T05:39:04.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:39:05.622 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:39:05.622 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:06.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803785 -lt 25769803785 2024-09-15T05:39:06.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:06.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574857 2024-09-15T05:39:06.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:06.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:39:06.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574857 2024-09-15T05:39:06.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:06.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574857 2024-09-15T05:39:06.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574857' 2024-09-15T05:39:06.042 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574857 2024-09-15T05:39:06.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:39:06.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574857 -lt 55834574857 2024-09-15T05:39:06.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:06.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345927 2024-09-15T05:39:06.493 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:06.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:39:06.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345927 2024-09-15T05:39:06.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:06.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345927 2024-09-15T05:39:06.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345927' 2024-09-15T05:39:06.495 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345927 2024-09-15T05:39:06.496 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:39:06.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345927 -lt 85899345927 2024-09-15T05:39:06.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:39:06.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:06.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:07.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:07.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:39:07.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:07.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:07.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:08.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:39:08.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:39:08.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:39:08.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:431: TEST_rep_read_unfound: ceph pg dump pgs 2024-09-15T05:39:08.453 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:39:08.453 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:39:03.723463+0000 0'0 30:20 [2,1] 2 [2,1] 2 0'0 2024-09-15T05:39:01.390222+0000 0'0 2024-09-15T05:39:01.390222+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:31:45.695735+0000 0 0 2024-09-15T05:39:08.453 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:39:08.453 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:39:08.454 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:39:08.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:433: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:08.464 INFO:tasks.workunit.client.0.smithi138.stderr:1+0 records in 2024-09-15T05:39:08.465 INFO:tasks.workunit.client.0.smithi138.stderr:1+0 records out 2024-09-15T05:39:08.465 INFO:tasks.workunit.client.0.smithi138.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 9.3189e-05 s, 87.9 MB/s 2024-09-15T05:39:08.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:434: TEST_rep_read_unfound: rados -p test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:08.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: get_primary test-pool myobject 2024-09-15T05:39:08.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test-pool 2024-09-15T05:39:08.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-15T05:39:08.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test-pool myobject 2024-09-15T05:39:08.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:39:08.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:436: TEST_rep_read_unfound: local primary=2 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: get_not_primary test-pool myobject 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test-pool 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=myobject 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test-pool myobject 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test-pool 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=myobject 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test-pool myobject 2024-09-15T05:39:08.924 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-15T05:39:09.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=2 2024-09-15T05:39:09.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test-pool myobject 2024-09-15T05:39:09.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 2)) | .[0]' 2024-09-15T05:39:09.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:437: TEST_rep_read_unfound: local other=1 2024-09-15T05:39:09.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:439: TEST_rep_read_unfound: dd if=/dev/urandom bs=8k count=1 of=td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:1+0 records in 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:1+0 records out 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:8192 bytes (8.2 kB, 8.0 KiB) copied, 6.5394e-05 s, 125 MB/s 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:440: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:09.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=2 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=2 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.2 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:39:09.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:39:09.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:39:09.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/2 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:10.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 2 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:39:10.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:39:10.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:39:10.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr:start osd.2 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/2/whoami 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T05:39:10.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:39:10.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:39:10.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:39:10.869 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:10.868+0000 7f6a6fe15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:10.871 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:10.870+0000 7f6a6fe15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:10.872 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:10.871+0000 7f6a6fe15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:10.874 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:10.873+0000 7f6a6fe15440 -1 Falling back to public interface 2024-09-15T05:39:11.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T05:39:11.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:39:11.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:0 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:11.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:39:11.509 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:11.508+0000 7f6a6fe15440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:39:11.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:11.515+0000 7f6a6fe15440 -1 journal do_read_entry(454656): bad header magic 2024-09-15T05:39:11.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:11.515+0000 7f6a6fe15440 -1 journal do_read_entry(454656): bad header magic 2024-09-15T05:39:11.521 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:11.520+0000 7f6a6fe15440 -1 osd.2 30 log_to_monitors true 2024-09-15T05:39:11.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:12.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:12.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:12.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:39:12.762 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:12.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:12.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:39:13.114 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 up in weight 1 up_from 34 up_thru 34 down_at 31 last_clean_interval [20,30) [v2:127.0.0.1:6818/447381,v1:127.0.0.1:6819/447381] [v2:127.0.0.1:6820/447381,v1:127.0.0.1:6821/447381] exists,up 13578e50-8db1-4087-b08c-c99925d41891 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:39:13.115 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:39:13.116 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:39:13.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:39:13.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:39:13.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:39:13.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:39:13.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:39:13.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:13.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:39:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803789 2024-09-15T05:39:13.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803789 2024-09-15T05:39:13.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803789' 2024-09-15T05:39:13.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:13.734 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:39:13.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574861 2024-09-15T05:39:13.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574861 2024-09-15T05:39:13.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803789 1-55834574861' 2024-09-15T05:39:13.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:13.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:39:13.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888067 2024-09-15T05:39:13.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888067 2024-09-15T05:39:13.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803789 1-55834574861 2-146028888067' 2024-09-15T05:39:13.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:13.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803789 2024-09-15T05:39:13.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:13.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:39:13.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803789 2024-09-15T05:39:13.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:13.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803789 2024-09-15T05:39:13.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803789' 2024-09-15T05:39:13.944 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.0 seq 25769803789 2024-09-15T05:39:13.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:14.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803787 -lt 25769803789 2024-09-15T05:39:14.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:39:15.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:39:15.297 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:15.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803789 -lt 25769803789 2024-09-15T05:39:15.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:15.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574861 2024-09-15T05:39:15.655 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:15.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:39:15.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574861 2024-09-15T05:39:15.656 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:15.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574861 2024-09-15T05:39:15.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574861' 2024-09-15T05:39:15.657 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.1 seq 55834574861 2024-09-15T05:39:15.657 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:39:16.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574861 -lt 55834574861 2024-09-15T05:39:16.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:16.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888067 2024-09-15T05:39:16.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:16.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:39:16.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888067 2024-09-15T05:39:16.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:16.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888067 2024-09-15T05:39:16.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-09-15T05:39:16.014 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.2 seq 146028888067 2024-09-15T05:39:16.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:39:16.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888067 -lt 146028888067 2024-09-15T05:39:16.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:39:16.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:16.434 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:17.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:39:17.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:17.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:17.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:39:17.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:17.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:17.416 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:441: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T05:39:17.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:39:17.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T05:39:18.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T05:39:18.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:18.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/CORRUPT 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:39:19.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:19.076 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:39:19.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T05:39:19.078 INFO:tasks.workunit.client.0.smithi138.stderr:start osd.1 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:39:19.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:39:19.105 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.102+0000 7fa44dc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:19.105 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.104+0000 7fa44dc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:19.107 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.105+0000 7fa44dc34440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:19.108 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.107+0000 7fa44dc34440 -1 Falling back to public interface 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:39:19.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:19.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:39:19.458 INFO:tasks.workunit.client.0.smithi138.stderr:0 2024-09-15T05:39:19.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:19.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:39:19.504 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.503+0000 7fa44dc34440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:39:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.511+0000 7fa44dc34440 -1 journal do_read_entry(716800): bad header magic 2024-09-15T05:39:19.512 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.511+0000 7fa44dc34440 -1 journal do_read_entry(716800): bad header magic 2024-09-15T05:39:19.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:19.515+0000 7fa44dc34440 -1 osd.1 35 log_to_monitors true 2024-09-15T05:39:19.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:20.869 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 up in weight 1 up_from 39 up_thru 31 down_at 36 last_clean_interval [13,35) [v2:127.0.0.1:6810/447935,v1:127.0.0.1:6811/447935] [v2:127.0.0.1:6812/447935,v1:127.0.0.1:6813/447935] exists,up 68424bb2-38bb-4cdd-8288-8b32bce58246 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:39:21.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:39:21.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:39:21.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:39:21.223 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:39:21.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:39:21.223 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:39:21.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:39:21.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:21.664 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:39:21.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803793 2024-09-15T05:39:21.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803793 2024-09-15T05:39:21.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793' 2024-09-15T05:39:21.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:21.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:39:22.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724547 2024-09-15T05:39:22.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724547 2024-09-15T05:39:22.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793 1-167503724547' 2024-09-15T05:39:22.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:22.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:39:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888071 2024-09-15T05:39:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888071 2024-09-15T05:39:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803793 1-167503724547 2-146028888071' 2024-09-15T05:39:22.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:22.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803793 2024-09-15T05:39:22.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:22.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:39:22.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803793 2024-09-15T05:39:22.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:22.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803793 2024-09-15T05:39:22.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803793' 2024-09-15T05:39:22.174 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.0 seq 25769803793 2024-09-15T05:39:22.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:22.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803790 -lt 25769803793 2024-09-15T05:39:22.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:39:23.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:39:23.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:23.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803793 -lt 25769803793 2024-09-15T05:39:23.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:23.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-167503724547 2024-09-15T05:39:23.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:23.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:39:23.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-167503724547 2024-09-15T05:39:23.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:23.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724547 2024-09-15T05:39:23.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 167503724547' 2024-09-15T05:39:23.879 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.1 seq 167503724547 2024-09-15T05:39:23.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:39:24.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724547 -lt 167503724547 2024-09-15T05:39:24.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:24.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888071 2024-09-15T05:39:24.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:24.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:39:24.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888071 2024-09-15T05:39:24.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:24.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888071 2024-09-15T05:39:24.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888071' 2024-09-15T05:39:24.233 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.2 seq 146028888071 2024-09-15T05:39:24.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:39:24.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888071 -lt 146028888071 2024-09-15T05:39:24.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:39:24.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:24.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:25.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:39:25.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:25.173 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:25.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:25.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:25.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:25.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:25.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:25.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:39:25.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:25.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:25.514 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:26.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:39:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:39:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:39:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:445: TEST_rep_read_unfound: sleep 5 2024-09-15T05:39:26.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:443: TEST_rep_read_unfound: timeout 30 rados -p test-pool get myobject td/osd-rep-recov-eio.sh/tmp 2024-09-15T05:39:31.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:447: TEST_rep_read_unfound: flush_pg_stats 2024-09-15T05:39:31.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:39:31.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:31.380 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:39:31.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803797 2024-09-15T05:39:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803797 2024-09-15T05:39:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797' 2024-09-15T05:39:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:31.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:39:31.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=167503724551 2024-09-15T05:39:31.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 167503724551 2024-09-15T05:39:31.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-167503724551' 2024-09-15T05:39:31.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:31.620 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:39:31.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888074 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888074 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-167503724551 2-146028888074' 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803797 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:31.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:39:31.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803797 2024-09-15T05:39:31.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:31.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803797 2024-09-15T05:39:31.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2024-09-15T05:39:31.726 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803797 2024-09-15T05:39:31.727 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:32.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803794 -lt 25769803797 2024-09-15T05:39:32.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:39:33.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:39:33.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:33.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803797 -lt 25769803797 2024-09-15T05:39:33.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:33.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-167503724551 2024-09-15T05:39:33.451 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:33.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:39:33.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-167503724551 2024-09-15T05:39:33.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:33.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=167503724551 2024-09-15T05:39:33.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 167503724551' 2024-09-15T05:39:33.454 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 167503724551 2024-09-15T05:39:33.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:39:33.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 167503724551 -lt 167503724551 2024-09-15T05:39:33.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:33.805 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888074 2024-09-15T05:39:33.805 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:33.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:39:33.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888074 2024-09-15T05:39:33.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:33.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888074 2024-09-15T05:39:33.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888074' 2024-09-15T05:39:33.807 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 146028888074 2024-09-15T05:39:33.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:39:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888075 -lt 146028888074 2024-09-15T05:39:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2024-09-15T05:39:34.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:448: TEST_rep_read_unfound: jq . 2024-09-15T05:39:34.512 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:39:34.523 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T05:39:34.523 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_ready": true, 2024-09-15T05:39:34.523 INFO:tasks.workunit.client.0.smithi138.stdout: "pg_stats": [ 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "pgid": "2.0", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "version": "30'1", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "reported_seq": 55, 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "reported_epoch": 40, 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "state": "active+recovery_unfound+degraded", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "last_fresh": "2024-09-15T05:39:26.028684+0000", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "last_change": "2024-09-15T05:39:26.229695+0000", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "last_active": "2024-09-15T05:39:26.028684+0000", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "last_peered": "2024-09-15T05:39:26.028684+0000", 2024-09-15T05:39:34.524 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean": "2024-09-15T05:39:20.888521+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_became_active": "2024-09-15T05:39:20.793226+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_became_peered": "2024-09-15T05:39:20.793226+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_unstale": "2024-09-15T05:39:26.028684+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_undegraded": "2024-09-15T05:39:26.028324+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_fullsized": "2024-09-15T05:39:26.028684+0000", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "mapping_epoch": 39, 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "log_start": "0'0", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_start": "0'0", 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "created": 26, 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "last_epoch_clean": 40, 2024-09-15T05:39:34.525 INFO:tasks.workunit.client.0.smithi138.stdout: "parent": "0.0", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "parent_split_bits": 0, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "last_scrub": "0'0", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "last_scrub_stamp": "2024-09-15T05:39:01.390222+0000", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "last_deep_scrub": "0'0", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "last_deep_scrub_stamp": "2024-09-15T05:39:01.390222+0000", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "last_clean_scrub_stamp": "2024-09-15T05:39:01.390222+0000", 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "objects_scrubbed": 0, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "log_size": 1, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "log_dups_size": 0, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "ondisk_log_size": 1, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "stats_invalid": false, 2024-09-15T05:39:34.526 INFO:tasks.workunit.client.0.smithi138.stdout: "dirty_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "omap_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "hitset_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "hitset_bytes_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "pin_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "manifest_stats_invalid": false, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "snaptrimq_len": 0, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "last_scrub_duration": 0, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "scrub_schedule": "periodic scrub scheduled @ 2024-09-16T17:28:09.213514+0000", 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "scrub_duration": 0, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "objects_trimmed": 0, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "snaptrim_duration": 0, 2024-09-15T05:39:34.527 INFO:tasks.workunit.client.0.smithi138.stdout: "stat_sum": { 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes": 8192, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects": 1, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_clones": 0, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_object_copies": 2, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing_on_primary": 1, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_missing": 0, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_degraded": 2, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_misplaced": 0, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_unfound": 1, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_dirty": 1, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_whiteouts": 0, 2024-09-15T05:39:34.528 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_read_kb": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write": 1, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_write_kb": 8, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_scrub_errors": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_shallow_scrub_errors": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_deep_scrub_errors": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_recovered": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_recovered": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_keys_recovered": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_omap": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_hit_set_archive": 0, 2024-09-15T05:39:34.529 INFO:tasks.workunit.client.0.smithi138.stdout: "num_bytes_hit_set_archive": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_kb": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_kb": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_promote": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_high": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_flush_mode_low": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_some": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_evict_mode_full": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_pinned": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_legacy_snapsets": 0, 2024-09-15T05:39:34.530 INFO:tasks.workunit.client.0.smithi138.stdout: "num_large_omap_objects": 0, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_manifest": 0, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_bytes": 0, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "num_omap_keys": 0, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "num_objects_repaired": 0 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: }, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "up": [ 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: 2, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: 1 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: "acting": [ 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: 2, 2024-09-15T05:39:34.531 INFO:tasks.workunit.client.0.smithi138.stdout: 1 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "avail_no_missing": [], 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "object_location_counts": [ 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: { 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "shards": "", 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "objects": 1 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: ], 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "blocked_by": [], 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "up_primary": 2, 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "acting_primary": 2, 2024-09-15T05:39:34.532 INFO:tasks.workunit.client.0.smithi138.stdout: "purged_snaps": [] 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stdout: } 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stdout: ] 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: ceph --format=json pg dump pgs 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: jq '.pg_stats | .[0].state' 2024-09-15T05:39:34.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:450: TEST_rep_read_unfound: grep -q recovery_unfound 2024-09-15T05:39:34.850 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:456: TEST_rep_read_unfound: objectstore_tool td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=1 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:34.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=1 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/osd-rep-recov-eio.sh TERM osd.1 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:39:34.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:39:35.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:39:35.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/osd-rep-recov-eio.sh 1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:35.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-rep-recov-eio.sh/1 myobject set-bytes td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:39:36.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:39:36.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:36.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:39:36.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:39:36.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T05:39:36.201 INFO:tasks.workunit.client.0.smithi138.stderr:start osd.1 2024-09-15T05:39:36.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:39:36.202 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/1/whoami 2024-09-15T05:39:36.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T05:39:36.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:39:36.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:39:36.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:39:36.225 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.224+0000 7f526260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:36.227 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.227+0000 7f526260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:36.229 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.228+0000 7f526260d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.230+0000 7f526260d440 -1 Falling back to public interface 2024-09-15T05:39:36.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:0 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:36.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:39:36.564 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.563+0000 7f526260d440 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2024-09-15T05:39:36.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.570+0000 7f526260d440 -1 journal do_read_entry(778240): bad header magic 2024-09-15T05:39:36.571 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.570+0000 7f526260d440 -1 journal do_read_entry(778240): bad header magic 2024-09-15T05:39:36.578 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:36.577+0000 7f526260d440 -1 osd.1 40 log_to_monitors true 2024-09-15T05:39:37.001 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 up in weight 1 up_from 44 up_thru 31 down_at 41 last_clean_interval [39,40) [v2:127.0.0.1:6810/448763,v1:127.0.0.1:6811/448763] [v2:127.0.0.1:6812/448763,v1:127.0.0.1:6813/448763] exists,up 68424bb2-38bb-4cdd-8288-8b32bce58246 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:39:37.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:39:37.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:39:37.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:39:37.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:39:37.613 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:39:37.613 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:39:37.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:39:37.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:37.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:39:37.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803800 2024-09-15T05:39:37.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803800 2024-09-15T05:39:37.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803800' 2024-09-15T05:39:37.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:37.719 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:39:37.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=188978561027 2024-09-15T05:39:37.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 188978561027 2024-09-15T05:39:37.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803800 1-188978561027' 2024-09-15T05:39:37.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:39:37.871 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:39:37.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=146028888078 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 146028888078 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803800 1-188978561027 2-146028888078' 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803800 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:37.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:39:37.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803800 2024-09-15T05:39:37.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:37.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803800 2024-09-15T05:39:37.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803800' 2024-09-15T05:39:37.978 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.0 seq 25769803800 2024-09-15T05:39:37.979 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:38.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803798 -lt 25769803800 2024-09-15T05:39:38.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:39:39.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:39:39.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:39:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803800 -lt 25769803800 2024-09-15T05:39:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:39.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-188978561027 2024-09-15T05:39:39.786 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:39.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:39:39.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-188978561027 2024-09-15T05:39:39.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:39.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=188978561027 2024-09-15T05:39:39.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 188978561027' 2024-09-15T05:39:39.789 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.1 seq 188978561027 2024-09-15T05:39:39.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:39:40.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 188978561027 -lt 188978561027 2024-09-15T05:39:40.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:39:40.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-146028888078 2024-09-15T05:39:40.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:39:40.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:39:40.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-146028888078 2024-09-15T05:39:40.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:39:40.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=146028888078 2024-09-15T05:39:40.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 146028888078' 2024-09-15T05:39:40.144 INFO:tasks.workunit.client.0.smithi138.stderr:waiting osd.2 seq 146028888078 2024-09-15T05:39:40.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:39:40.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 146028888078 -lt 146028888078 2024-09-15T05:39:40.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:39:40.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:40.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:40.974 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:40.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:40.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:41.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:39:41.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:41.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:41.320 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:41.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:39:41.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:39:41.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:39:41.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:39:41.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:39:41.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:39:41.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:39:41.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:41.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:41.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:41.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:41.893 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:41.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:41.894 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:42.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:39:42.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:42.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:42.257 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:39:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:39:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:39:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:39:42.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:39:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:39:42.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:39:42.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:39:42.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:39:43.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1280 2024-09-15T05:39:43.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1280 '!=' null 2024-09-15T05:39:43.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:39:43.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:39:43.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:39:43.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:39:43.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:39:43.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:39:43.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:39:43.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:39:43.788 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:39:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:39:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:39:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:39:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:458: TEST_rep_read_unfound: wait 2024-09-15T05:39:44.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:460: TEST_rep_read_unfound: cmp td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/tmp 2024-09-15T05:39:44.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:39:44.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:39:44.255 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:39:44.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:39:44.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:39:44.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:39:44.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:39:44.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:39:44.365 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:39:44.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:39:44.366 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:39:44.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:39:44.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:39:44.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:39:44.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:39:44.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:39:44.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:39:44.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:39:44.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:39:44.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:39:44.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:39:44.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:44.383 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:44.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:39:44.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:39:44.386 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:39:44.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:39:44.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:39:44.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:39:44.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:39:44.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:39:44.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:39:44.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:39:44.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:39:44.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:39:44.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:39:44.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:39:44.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:44.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:39:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:39:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:39:44.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:39:44.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:44.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:44.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.400988 2024-09-15T05:39:44.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:39:44.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:39:44.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:39:44.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-09-15T05:39:44.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:44.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-09-15T05:39:44.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:39:44.995 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:39:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:39:44.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get fsid 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:39:45.071 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-mon.a.asok 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:39:45.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.400988/ceph-mon.a.asok config get mon_host 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-09-15T05:39:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:39:45.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:39:45.990 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:39:45.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:39:46.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-09-15T05:39:46.298 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:39:46.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-09-15T05:39:46.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-09-15T05:39:46.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:39:46.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:39:46.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:39:46.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:39:46.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5bdf82f4-b2d3-43fc-bea0-bab5ac801454 2024-09-15T05:39:46.316 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 5bdf82f4-b2d3-43fc-bea0-bab5ac801454 2024-09-15T05:39:46.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5bdf82f4-b2d3-43fc-bea0-bab5ac801454' 2024-09-15T05:39:46.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:39:46.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAic+ZmgRekExAA4/pQVjbdl/nYHNFmWRtruw== 2024-09-15T05:39:46.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAic+ZmgRekExAA4/pQVjbdl/nYHNFmWRtruw=="}' 2024-09-15T05:39:46.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5bdf82f4-b2d3-43fc-bea0-bab5ac801454 -i td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:39:46.627 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:39:46.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-09-15T05:39:46.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAic+ZmgRekExAA4/pQVjbdl/nYHNFmWRtruw== --osd-uuid 5bdf82f4-b2d3-43fc-bea0-bab5ac801454 2024-09-15T05:39:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.663+0000 7f51f9324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:46.674 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.673+0000 7f51f9324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:46.675 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.674+0000 7f51f9324440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:46.675 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.674+0000 7f51f9324440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:39:46.675 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.674+0000 7f51f9324440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:39:46.675 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.674+0000 7f51f9324440 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-09-15T05:39:46.714 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:46.713+0000 7f51f9324440 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-09-15T05:39:52.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-09-15T05:39:52.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:39:52.002 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:39:52.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:39:52.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:39:52.487 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:39:52.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:39:52.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:39:52.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:39:52.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:39:52.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:39:52.511 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:52.510+0000 7fcfe2c0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:52.513 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:52.512+0000 7fcfe2c0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:52.514 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:52.513+0000 7fcfe2c0e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:39:52.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:39:52.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:39:52.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:39:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:39:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:39:52.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:52.836 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:39:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:39:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:52.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:39:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:53.030+0000 7fcfe2c0e440 -1 Falling back to public interface 2024-09-15T05:39:53.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:54.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:54.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:54.199 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:39:54.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:39:54.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:54.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:39:54.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:55.522 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:39:55.521+0000 7fcfe2c0e440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:39:55.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:55.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:55.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:39:55.560 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:39:55.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:55.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:39:56.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:57.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:57.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:57.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:39:57.117 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:39:57.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:57.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:39:57.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:58.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:58.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:58.526 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:39:58.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:39:58.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:58.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:39:58.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:39:59.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:39:59.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:39:59.925 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:39:59.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:39:59.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:39:59.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:00.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:01.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:01.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:01.310 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:40:01.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:40:01.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:01.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:01.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:02.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:02.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:02.677 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:40:02.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:40:02.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:02.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:03.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:04.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:04.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:04.034 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:40:04.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:40:04.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:04.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:04.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:05.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:05.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:05.398 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:40:05.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:40:05.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:05.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:05.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:06.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:06.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:06.794 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:40:06.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:40:06.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:06.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:07.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:08.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:08.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:08.305 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:40:08.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:40:08.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:08.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:08.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:09.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:09.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:09.731 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:40:09.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:40:09.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:09.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:10.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:11.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:11.363 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:40:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:40:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:11.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:11.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:12.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:12.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:12.921 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:40:12.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:40:12.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:12.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:13.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:14.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:14.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:14.675 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:40:14.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:40:14.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:14.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:15.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:16.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:16.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:16.539 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:40:16.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:40:16.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:16.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:17.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:18.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:18.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:18.251 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:40:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:40:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:18.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:40:18.740 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/450495,v1:127.0.0.1:6803/450495] [v2:127.0.0.1:6804/450495,v1:127.0.0.1:6805/450495] exists,up 5bdf82f4-b2d3-43fc-bea0-bab5ac801454 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:40:18.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:40:18.745 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:40:18.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62cdc3ac-0fcc-4126-889a-503e7fb1f292 2024-09-15T05:40:18.747 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 62cdc3ac-0fcc-4126-889a-503e7fb1f292 2024-09-15T05:40:18.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 62cdc3ac-0fcc-4126-889a-503e7fb1f292' 2024-09-15T05:40:18.748 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:40:18.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBCc+ZmeRM2LRAALjjQ6eDpNgWa377JpRhYdw== 2024-09-15T05:40:18.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBCc+ZmeRM2LRAALjjQ6eDpNgWa377JpRhYdw=="}' 2024-09-15T05:40:18.759 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62cdc3ac-0fcc-4126-889a-503e7fb1f292 -i td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:40:19.593 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:40:19.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-09-15T05:40:19.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBCc+ZmeRM2LRAALjjQ6eDpNgWa377JpRhYdw== --osd-uuid 62cdc3ac-0fcc-4126-889a-503e7fb1f292 2024-09-15T05:40:19.632 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.631+0000 7f2513608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:19.633 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.633+0000 7f2513608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:19.634 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.634+0000 7f2513608440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:19.635 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.634+0000 7f2513608440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:40:19.635 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.634+0000 7f2513608440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:40:19.635 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.634+0000 7f2513608440 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-09-15T05:40:19.718 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:19.717+0000 7f2513608440 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-09-15T05:40:25.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-09-15T05:40:25.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:40:25.632 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:40:25.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:40:25.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:40:26.116 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:40:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:40:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:40:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:40:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:40:26.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:40:26.140 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:26.139+0000 7fadada2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:26.142 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:26.142+0000 7fadada2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:26.143 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:26.143+0000 7fadada2d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:26.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:40:26.472 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:40:26.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:26.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:26.668 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:26.667+0000 7fadada2d440 -1 Falling back to public interface 2024-09-15T05:40:26.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:27.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:27.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:27.878 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:40:27.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:40:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:27.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:28.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:29.159 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:29.159+0000 7fadada2d440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:40:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:29.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:29.233 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:40:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:40:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:29.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:29.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:30.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:30.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:30.727 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:40:30.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:40:30.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:30.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:31.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:32.132 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:40:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:40:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:32.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:33.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:33.590 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:40:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:40:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:33.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:34.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:35.050 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:40:35.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:40:35.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:35.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:35.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:36.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:36.466 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:40:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:40:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:36.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:36.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:37.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:37.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:37.966 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:40:37.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:40:37.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:37.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:38.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:39.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:39.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:39.471 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:40:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:40:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:39.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:39.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:40.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:40.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:40.837 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:40:40.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:40:40.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:40.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:41.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:42.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:42.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:42.277 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:40:42.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:40:42.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:42.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:42.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:43.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:43.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:43.691 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:40:43.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:40:43.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:43.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:44.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:45.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:45.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:45.239 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:40:45.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:40:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:45.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:46.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:47.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:47.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:47.051 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:40:47.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:40:47.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:47.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:47.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:48.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:48.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:48.752 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:40:48.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:40:48.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:48.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:49.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:50.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:50.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:50.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:40:50.680 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:40:50.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:50.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:51.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:52.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:52.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:52.811 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:40:52.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:40:52.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:52.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:53.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:40:54.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:40:54.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:40:54.288 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:40:54.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:40:54.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:40:54.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:40:54.718 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/452263,v1:127.0.0.1:6811/452263] [v2:127.0.0.1:6812/452263,v1:127.0.0.1:6813/452263] exists,up 62cdc3ac-0fcc-4126-889a-503e7fb1f292 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:40:54.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:40:54.723 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:40:54.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=118c7dd6-627b-484c-9804-72e1320c7e27 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 118c7dd6-627b-484c-9804-72e1320c7e27' 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 118c7dd6-627b-484c-9804-72e1320c7e27 2024-09-15T05:40:54.725 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:40:54.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBmc+Zm493qKxAAaEKeU5NdvkkhXkmFqW6HGA== 2024-09-15T05:40:54.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBmc+Zm493qKxAAaEKeU5NdvkkhXkmFqW6HGA=="}' 2024-09-15T05:40:54.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 118c7dd6-627b-484c-9804-72e1320c7e27 -i td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:40:55.232 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:40:55.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-09-15T05:40:55.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBmc+Zm493qKxAAaEKeU5NdvkkhXkmFqW6HGA== --osd-uuid 118c7dd6-627b-484c-9804-72e1320c7e27 2024-09-15T05:40:55.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.269+0000 7f92cb832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:55.271 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.270+0000 7f92cb832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:55.272 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.271+0000 7f92cb832440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:40:55.273 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.272+0000 7f92cb832440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:40:55.273 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.272+0000 7f92cb832440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:40:55.273 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.272+0000 7f92cb832440 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-09-15T05:40:55.372 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:40:55.372+0000 7f92cb832440 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-09-15T05:41:01.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-09-15T05:41:01.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:41:01.342 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:41:01.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:41:01.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:41:01.842 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:41:01.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:41:01.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:41:01.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:41:01.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:41:01.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:41:01.866 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:41:01.865+0000 7f63edf27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:41:01.869 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:41:01.867+0000 7f63edf27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:41:01.870 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:41:01.868+0000 7f63edf27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:41:02.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:02.195 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:41:02.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:41:02.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:02.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:02.386 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:41:02.384+0000 7f63edf27440 -1 Falling back to public interface 2024-09-15T05:41:02.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:03.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:03.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:03.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:41:03.582 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:41:03.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:03.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:03.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:04.895 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:41:04.894+0000 7f63edf27440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:41:04.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:04.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:04.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:41:04.941 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:41:04.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:04.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:05.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:06.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:06.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:06.478 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:41:06.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:41:06.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:06.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:06.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:07.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:07.980 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:07.980 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:41:07.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:41:07.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:07.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:08.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:09.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:09.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:09.472 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:41:09.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:41:09.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:09.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:10.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:11.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:11.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:11.015 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:41:11.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:41:11.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:11.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:11.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:12.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:12.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:12.378 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:41:12.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:41:12.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:12.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:12.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:13.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:13.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:13.887 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:41:13.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:41:13.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:13.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:14.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:15.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:15.267 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:41:15.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:41:15.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:15.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:15.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:16.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:16.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:16.793 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:41:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:41:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:17.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:18.199 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:41:18.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:41:18.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:18.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:18.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:19.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:19.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:19.905 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:41:19.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:41:19.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:19.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:20.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:21.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:21.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:21.873 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:41:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:41:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:21.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:22.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:23.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:23.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:23.964 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:41:23.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:41:23.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:23.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:24.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:25.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:25.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:25.822 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:41:25.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:41:25.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:25.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:26.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:27.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:27.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:27.619 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:41:27.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:41:27.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:27.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:28.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:41:29.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:41:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:41:29.404 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:41:29.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:41:29.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:41:29.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:41:29.778 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/454043,v1:127.0.0.1:6819/454043] [v2:127.0.0.1:6820/454043,v1:127.0.0.1:6821/454043] exists,up 118c7dd6-627b-484c-9804-72e1320c7e27 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:41:29.780 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:41:29.781 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:41:29.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:41:29.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:41:29.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:41:29.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:41:29.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:41:29.866 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:41:30.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:41:30.406 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:41:30.406 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:41:30.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:41:30.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:30.406 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:41:30.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803797 2024-09-15T05:41:30.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803797 2024-09-15T05:41:30.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797' 2024-09-15T05:41:30.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:30.625 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:41:30.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574860 2024-09-15T05:41:30.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574860 2024-09-15T05:41:30.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-55834574860' 2024-09-15T05:41:30.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:30.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:41:31.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345928 2024-09-15T05:41:31.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345928 2024-09-15T05:41:31.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803797 1-55834574860 2-85899345928' 2024-09-15T05:41:31.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:31.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803797 2024-09-15T05:41:31.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:31.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:41:31.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803797 2024-09-15T05:41:31.011 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803797 2024-09-15T05:41:31.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2024-09-15T05:41:31.012 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803797 2024-09-15T05:41:31.012 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:41:31.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803795 -lt 25769803797 2024-09-15T05:41:31.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:41:32.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:41:32.560 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:41:32.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803797 -lt 25769803797 2024-09-15T05:41:32.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:32.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574860 2024-09-15T05:41:32.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:32.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:41:32.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574860 2024-09-15T05:41:32.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:32.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574860 2024-09-15T05:41:32.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574860' 2024-09-15T05:41:32.919 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574860 2024-09-15T05:41:32.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:41:33.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574860 -lt 55834574860 2024-09-15T05:41:33.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:33.271 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345928 2024-09-15T05:41:33.271 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:33.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:41:33.272 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345928 2024-09-15T05:41:33.273 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:33.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345928 2024-09-15T05:41:33.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345928' 2024-09-15T05:41:33.274 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345928 2024-09-15T05:41:33.274 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:41:33.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345928 -lt 85899345928 2024-09-15T05:41:33.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:41:33.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:33.626 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:41:34.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:41:34.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:41:34.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:41:34.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:34.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 8 2024-09-15T05:41:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:41:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:41:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:41:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:41:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:41:35.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:41:35.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:41:35.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:41:35.362 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:41:35.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:35.363 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 8 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:41:35.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:41:35.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:41:35.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:41:36.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T05:41:36.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T05:41:36.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T05:41:36.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:41:36.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T05:41:36.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:41:36.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:41:36.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:41:36.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-15T05:41:36.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:41:36.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:36.865 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:37.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-15T05:41:37.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:41:37.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:41:37.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-09-15T05:41:37.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-09-15T05:41:37.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-09-15T05:41:37.866 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test-pool' already exists 2024-09-15T05:41:37.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:41:38.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-09-15T05:41:38.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:41:38.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:41:38.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:41:38.879 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:41:38.880 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:41:38.880 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:41:38.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:41:38.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:41:38.880 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:41:38.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:41:38.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:41:38.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:41:38.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:41:38.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:41:38.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:39.322 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:41:39.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=25769803801 2024-09-15T05:41:39.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 25769803801 2024-09-15T05:41:39.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801' 2024-09-15T05:41:39.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:39.445 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:41:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574864 2024-09-15T05:41:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574864 2024-09-15T05:41:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801 1-55834574864' 2024-09-15T05:41:39.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:41:39.553 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:41:39.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345932 2024-09-15T05:41:39.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345932 2024-09-15T05:41:39.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-25769803801 1-55834574864 2-85899345932' 2024-09-15T05:41:39.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-25769803801 2024-09-15T05:41:39.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:39.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:41:39.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-25769803801 2024-09-15T05:41:39.662 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:39.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=25769803801 2024-09-15T05:41:39.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-09-15T05:41:39.663 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 25769803801 2024-09-15T05:41:39.663 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:41:40.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 25769803801 -lt 25769803801 2024-09-15T05:41:40.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:40.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574864 2024-09-15T05:41:40.036 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:40.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:41:40.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574864 2024-09-15T05:41:40.037 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:40.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574864 2024-09-15T05:41:40.038 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574864 2024-09-15T05:41:40.039 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574864' 2024-09-15T05:41:40.039 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:41:40.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574863 -lt 55834574864 2024-09-15T05:41:40.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:41:41.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:41:41.443 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:41:41.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574864 -lt 55834574864 2024-09-15T05:41:41.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:41:41.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345932 2024-09-15T05:41:41.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:41:41.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:41:41.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345932 2024-09-15T05:41:41.803 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:41:41.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345932 2024-09-15T05:41:41.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-09-15T05:41:41.804 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345932 2024-09-15T05:41:41.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:41:42.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345932 -lt 85899345932 2024-09-15T05:41:42.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:41:42.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:42.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:42.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:41:42.623 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:41:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:41:42.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:41:42.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:41:42.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:41:42.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:41:42.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:41:42.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:41:42.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:41:42.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:41:42.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:41:43.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:41:43.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:41:43.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:41:43.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-15T05:41:43.772 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:41:43.772 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009705+0000 0'0 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:21:08.245520+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009859+0000 0'0 24:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:54:35.880247+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.455703+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:04:08.629847+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009609+0000 0'0 24:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:05:47.399553+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:38.899575+0000 0'0 25:10 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T05:41:37.697751+0000 0'0 2024-09-15T05:41:37.697751+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:00:17.842128+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009635+0000 0'0 24:26 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:13:26.435775+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.514215+0000 0'0 24:38 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:20:43.705255+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.455725+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:57:30.273939+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009958+0000 0'0 24:26 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:36:19.031196+0000 0 0 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:41:43.773 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:41:43.774 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:41:43.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-09-15T05:41:43.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:41:43.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-09-15T05:41:43.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-09-15T05:41:43.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-09-15T05:41:43.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-09-15T05:41:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:41:43.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-09-15T05:41:43.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-09-15T05:41:43.908 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:0' 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-09-15T05:41:44.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:41:44.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:41:44.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:41:44.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:41:44.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:41:44.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:41:44.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-09-15T05:41:45.228 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already down. 2024-09-15T05:41:45.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-09-15T05:41:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. 2024-09-15T05:41:45.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009705+0000 0'0 25:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:21:08.245520+0000 0 0 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009859+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T13:54:35.880247+0000 0 0 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.455703+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:04:08.629847+0000 0 0 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009609+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T11:05:47.399553+0000 0 0 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 active+clean 2024-09-15T05:41:38.899575+0000 26'1 26:11 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T05:41:37.697751+0000 0'0 2024-09-15T05:41:37.697751+0000 0 0 periodic scrub scheduled @ 2024-09-16T06:00:17.842128+0000 0 0 2024-09-15T05:41:46.129 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009635+0000 0'0 25:27 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:13:26.435775+0000 0 0 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-09-15T05:41:31.514215+0000 0'0 25:39 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T16:20:43.705255+0000 0 0 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.455725+0000 0'0 25:14 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:57:30.273939+0000 0 0 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:41:31.009958+0000 0'0 25:27 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:36:19.031196+0000 0 0 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:41:46.130 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:41:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-09-15T05:41:46.140 INFO:tasks.workunit.client.0.smithi138.stderr:4+0 records in 2024-09-15T05:41:46.141 INFO:tasks.workunit.client.0.smithi138.stderr:4+0 records out 2024-09-15T05:41:46.141 INFO:tasks.workunit.client.0.smithi138.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 9.7867e-05 s, 41.9 MB/s 2024-09-15T05:41:46.141 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-09-15T05:41:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:46.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:46.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:47.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:47.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.113 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:48.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:48.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:49.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:49.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:50.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:50.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:51.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:51.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:52.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:52.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:53.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:53.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:54.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:54.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.823 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:55.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:55.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:56.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:56.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:57.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:57.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:58.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:41:58.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:41:59.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=0 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj75 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-15T05:41:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=2 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:41:59.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-09-15T05:41:59.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:41:59.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-09-15T05:41:59.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:41:59.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.2.asok 2024-09-15T05:41:59.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:41:59.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:41:59.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.2 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.2.asok 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:41:59.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.2.asok injectdataerr test-pool obj75 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2365: inject_eio: local pooltype=rep 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2366: inject_eio: shift 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2367: inject_eio: local which=data 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2368: inject_eio: shift 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2369: inject_eio: local poolname=test-pool 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2370: inject_eio: shift 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2371: inject_eio: local objname=obj75 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2372: inject_eio: shift 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2373: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:41:59.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2374: inject_eio: shift 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2375: inject_eio: local shard_id=1 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2376: inject_eio: shift 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: get_osds test-pool obj75 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-09-15T05:41:59.619 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: initial_osds=('2' '1') 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2378: inject_eio: local -a initial_osds 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2379: inject_eio: local osd_id=1 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2380: inject_eio: '[' rep '!=' ec ']' 2024-09-15T05:41:59.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2381: inject_eio: shard_id= 2024-09-15T05:41:59.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-09-15T05:41:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2383: inject_eio: type=bluestore 2024-09-15T05:41:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2384: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-09-15T05:41:59.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:41:59.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.400988/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-09-15T05:42:00.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-15T05:42:00.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2385: inject_eio: local loop=0 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2387: inject_eio: grep -q Invalid 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: get_asok_path osd.1 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:00.052 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:42:00.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.400988/ceph-osd.1.asok 2024-09-15T05:42:00.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: CEPH_ARGS= 2024-09-15T05:42:00.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2386: inject_eio: ceph --admin-daemon /tmp/ceph-asok.400988/ceph-osd.1.asok injectdataerr test-pool obj75 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:42:00.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:42:00.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:42:00.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-09-15T05:42:00.138 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=f8d4f6e2-8037-4736-acc3-09b9147ba5da --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.400988/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:42:00.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:42:00.158 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:00.157+0000 7f0a5d24e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:00.166 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:00.166+0000 7f0a5d24e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:00.168 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:00.167+0000 7f0a5d24e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:00.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:42:00.484 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:42:00.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:00.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:00.684+0000 7f0a5d24e440 -1 Falling back to public interface 2024-09-15T05:42:00.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:01.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:01.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:01.847 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:42:01.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:42:01.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:01.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:02.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:03.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:03.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:03.229 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:42:03.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:42:03.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:03.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:03.570 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:03.569+0000 7f0a5d24e440 -1 osd.0 26 log_to_monitors true 2024-09-15T05:42:03.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:04.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:04.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:04.579 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:42:04.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:42:04.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:04.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:05.332 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/458010,v1:127.0.0.1:6803/458010] [v2:127.0.0.1:6804/458010,v1:127.0.0.1:6805/458010] exists,up 5bdf82f4-b2d3-43fc-bea0-bab5ac801454 2024-09-15T05:42:05.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:42:05.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:42:05.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:42:05.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-09-15T05:42:06.176 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. 2024-09-15T05:42:06.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-09-15T05:42:21.188 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-09-15T05:42:21.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-09-15T05:42:21.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-09-15T05:42:21.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-09-15T05:42:21.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-09-15T05:42:21.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-09-15T05:42:21.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-09-15T05:42:21.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-09-15T05:42:21.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-09-15T05:42:21.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-09-15T05:42:21.543 INFO:tasks.workunit.client.0.smithi138.stdout:active+recovery_unfound+degraded 2024-09-15T05:42:21.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-09-15T05:42:21.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-09-15T05:42:21.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-09-15T05:42:21.932 INFO:tasks.workunit.client.0.smithi138.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.023444+0000 0'0 39:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T08:38:02.415386+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.932509+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:26:26.163151+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.932490+0000 0'0 40:56 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:14:42.033918+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.932591+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T12:52:16.709741+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-09-15T05:42:11.590307+0000 30'101 40:350 [2,1,0] 2 [2,1,0] 2 0'0 2024-09-15T05:41:37.697751+0000 0'0 2024-09-15T05:41:37.697751+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:17:31.370790+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.090508+0000 0'0 39:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:42:30.818998+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.739841+0000 0'0 40:59 [0,1,2] 0 [0,1,2] 0 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T14:58:57.471655+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.933489+0000 0'0 40:56 [2,0,1] 2 [2,0,1] 2 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T07:18:44.219909+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-09-15T05:42:07.932579+0000 0'0 40:69 [1,0,2] 1 [1,0,2] 1 0'0 2024-09-15T05:39:46.190839+0000 0'0 2024-09-15T05:39:46.190839+0000 0 0 periodic scrub scheduled @ 2024-09-16T15:23:49.895406+0000 0 0 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:42:21.933 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:42:21.934 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pgs 2024-09-15T05:42:21.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-09-15T05:42:21.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-09-15T05:42:22.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:27.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-09-15T05:42:27.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-09-15T05:42:27.249 INFO:tasks.workunit.client.0.smithi138.stderr:pg has no unfound objects 2024-09-15T05:42:27.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-09-15T05:42:27.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:42:27.258 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:42:27.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:42:27.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:42:27.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:42:27.259 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:42:27.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:42:27.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:42:27.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:42:27.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:42:27.697 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:42:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=137438953479 2024-09-15T05:42:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 137438953479 2024-09-15T05:42:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479' 2024-09-15T05:42:27.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:42:27.805 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:42:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=55834574875 2024-09-15T05:42:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 55834574875 2024-09-15T05:42:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479 1-55834574875' 2024-09-15T05:42:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:42:27.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-137438953479 1-55834574875 2-85899345943' 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-137438953479 2024-09-15T05:42:28.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:42:28.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:42:28.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-137438953479 2024-09-15T05:42:28.042 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:42:28.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=137438953479 2024-09-15T05:42:28.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 137438953479' 2024-09-15T05:42:28.043 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 137438953479 2024-09-15T05:42:28.043 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:42:28.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953477 -lt 137438953479 2024-09-15T05:42:28.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:42:29.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:42:29.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:42:29.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 137438953479 -lt 137438953479 2024-09-15T05:42:29.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:42:29.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-55834574875 2024-09-15T05:42:29.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:42:29.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:42:29.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-55834574875 2024-09-15T05:42:29.783 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:42:29.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=55834574875 2024-09-15T05:42:29.784 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 55834574875 2024-09-15T05:42:29.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-09-15T05:42:29.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:42:30.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 55834574876 -lt 55834574875 2024-09-15T05:42:30.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:42:30.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-85899345943 2024-09-15T05:42:30.143 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:42:30.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:42:30.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-85899345943 2024-09-15T05:42:30.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:42:30.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-15T05:42:30.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 85899345943' 2024-09-15T05:42:30.145 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 85899345943 2024-09-15T05:42:30.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:42:30.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345944 -lt 85899345943 2024-09-15T05:42:30.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:42:30.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:42:30.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:42:30.977 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:42:31.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-09-15T05:42:31.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:42:31.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:42:31.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:42:31.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-09-15T05:42:31.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:42:31.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:42:31.800 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-09-15T05:42:31.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:31.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-09-15T05:42:31.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:31.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-09-15T05:42:31.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:31.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-09-15T05:42:31.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:31.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-09-15T05:42:31.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:31.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-09-15T05:42:32.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-09-15T05:42:32.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-09-15T05:42:32.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-09-15T05:42:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-09-15T05:42:32.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-09-15T05:42:32.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-09-15T05:42:32.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-09-15T05:42:32.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-09-15T05:42:32.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-09-15T05:42:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-09-15T05:42:32.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-09-15T05:42:32.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-09-15T05:42:32.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-09-15T05:42:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-09-15T05:42:32.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-09-15T05:42:32.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-09-15T05:42:32.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-09-15T05:42:32.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-09-15T05:42:32.563 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-09-15T05:42:32.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-09-15T05:42:32.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-09-15T05:42:32.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-09-15T05:42:32.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-09-15T05:42:32.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-09-15T05:42:32.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-09-15T05:42:32.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-09-15T05:42:32.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-09-15T05:42:32.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-09-15T05:42:32.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-09-15T05:42:32.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-09-15T05:42:32.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-09-15T05:42:32.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-09-15T05:42:32.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-09-15T05:42:32.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:32.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:32.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-09-15T05:42:32.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-09-15T05:42:33.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-09-15T05:42:33.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-09-15T05:42:33.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-09-15T05:42:33.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-09-15T05:42:33.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-09-15T05:42:33.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-09-15T05:42:33.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-09-15T05:42:33.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-09-15T05:42:33.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-09-15T05:42:33.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-09-15T05:42:33.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-09-15T05:42:33.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-09-15T05:42:33.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-09-15T05:42:33.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-09-15T05:42:33.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-09-15T05:42:33.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-09-15T05:42:33.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-09-15T05:42:33.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-09-15T05:42:33.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-09-15T05:42:33.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-09-15T05:42:33.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-09-15T05:42:33.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-09-15T05:42:33.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-09-15T05:42:33.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-09-15T05:42:33.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-09-15T05:42:33.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.740 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-09-15T05:42:33.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-09-15T05:42:33.767 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-09-15T05:42:33.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-09-15T05:42:33.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-09-15T05:42:33.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-09-15T05:42:33.876 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-09-15T05:42:33.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-09-15T05:42:33.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.957 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-09-15T05:42:33.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:33.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:33.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-09-15T05:42:33.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.009 INFO:tasks.workunit.client.0.smithi138.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-09-15T05:42:34.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-09-15T05:42:34.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-09-15T05:42:34.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-09-15T05:42:34.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-09-15T05:42:34.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-09-15T05:42:34.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-09-15T05:42:34.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-09-15T05:42:34.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-09-15T05:42:34.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-09-15T05:42:34.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-09-15T05:42:34.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-09-15T05:42:34.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-09-15T05:42:34.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-09-15T05:42:34.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-09-15T05:42:34.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-09-15T05:42:34.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-09-15T05:42:34.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-09-15T05:42:34.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-09-15T05:42:34.508 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-09-15T05:42:34.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-09-15T05:42:34.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-09-15T05:42:34.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-09-15T05:42:34.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-09-15T05:42:34.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-09-15T05:42:34.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-09-15T05:42:34.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-09-15T05:42:34.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-09-15T05:42:34.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-09-15T05:42:34.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-09-15T05:42:34.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-09-15T05:42:35.221 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test-pool' does not exist 2024-09-15T05:42:35.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-09-15T05:42:35.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:42:35.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:42:35.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:42:35.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:42:35.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:42:35.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:42:35.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:42:35.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:42:35.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:42:35.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:42:35.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:42:35.341 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:42:35.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:42:35.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:42:35.344 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:42:35.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:42:35.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:42:35.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:35.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:42:35.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-rep-recov-eio.sh 0 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:42:35.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:42:35.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:42:35.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:42:35.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:42:35.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:42:35.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:42:35.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:42:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:42:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:42:35.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:42:35.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:42:35.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:42:35.376 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:42:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T05:42:35.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-09-15T05:42:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:42:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:35.379 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.400988 2024-09-15T05:42:35.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.400988 2024-09-15T05:42:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:42:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:42:35.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T05:42:35.381 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T05:42:35.381 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T05:42:35.446 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-09-15T05:42:35.447 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-09-15T05:42:35.509 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T05:42:35.513 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/osd-reuse-id 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-09-15T05:42:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none ' 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-09-15T05:42:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:42:35.517 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:42:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:42:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:42:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:42:35.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:42:35.518 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:42:35.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:42:35.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:42:35.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:42:35.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:42:35.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:42:35.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:42:35.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:42:35.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:42:35.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:42:35.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:42:35.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-15T05:42:35.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:42:35.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:35.524 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:35.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.461257 2024-09-15T05:42:35.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:42:35.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:42:35.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-09-15T05:42:35.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:42:35.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:35.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:35.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.461257 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:42:35.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:42:35.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:42:35.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-09-15T05:42:35.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:35.902 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:42:35.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:42:36.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:42:36.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:42:36.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:42:36.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.461257/ceph-mon.a.asok 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:42:36.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.461257/ceph-mon.a.asok config get fsid 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:42:36.305 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.461257/ceph-mon.a.asok 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:42:36.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.461257/ceph-mon.a.asok config get mon_host 2024-09-15T05:42:36.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-09-15T05:42:36.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:37.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:42:37.225 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:42:37.225 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:42:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:42:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:42:37.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:42:37.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:42:37.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-09-15T05:42:37.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:42:37.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8edde77f-2569-4512-85cb-350ceb4db2aa 2024-09-15T05:42:37.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8edde77f-2569-4512-85cb-350ceb4db2aa' 2024-09-15T05:42:37.254 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 8edde77f-2569-4512-85cb-350ceb4db2aa 2024-09-15T05:42:37.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:42:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNc+ZmDL+sDxAArGeHvjQ3/+9P82LFzywiRA== 2024-09-15T05:42:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNc+ZmDL+sDxAArGeHvjQ3/+9P82LFzywiRA=="}' 2024-09-15T05:42:37.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8edde77f-2569-4512-85cb-350ceb4db2aa -i td/osd-reuse-id/0/new.json 2024-09-15T05:42:37.567 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:42:37.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-09-15T05:42:37.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNc+ZmDL+sDxAArGeHvjQ3/+9P82LFzywiRA== --osd-uuid 8edde77f-2569-4512-85cb-350ceb4db2aa 2024-09-15T05:42:37.599 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.597+0000 7f46dca1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:37.612 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.610+0000 7f46dca1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:37.613 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.611+0000 7f46dca1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:37.614 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.612+0000 7f46dca1e440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-15T05:42:37.614 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.612+0000 7f46dca1e440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-15T05:42:37.614 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.612+0000 7f46dca1e440 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-09-15T05:42:37.652 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:37.651+0000 7f46dca1e440 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-09-15T05:42:42.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-09-15T05:42:42.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:42:42.965 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:42:42.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:42:42.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:42:43.442 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:42:43.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:42:43.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:42:43.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:42:43.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:42:43.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:42:43.463 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:43.462+0000 7fc96e212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:43.465 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:43.464+0000 7fc96e212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:43.466 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:43.466+0000 7fc96e212440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:42:43.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:42:43.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:42:43.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:42:43.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:42:43.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:42:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:42:43.793 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:42:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:43.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:43.981 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:43.980+0000 7fc96e212440 -1 Falling back to public interface 2024-09-15T05:42:44.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:45.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:45.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:45.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:42:45.144 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:42:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:45.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:46.444 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:42:46.443+0000 7fc96e212440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:42:46.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:46.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:46.499 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:42:46.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:42:46.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:46.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:47.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:48.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:48.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:48.023 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:42:48.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:42:48.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:48.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:48.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:49.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:49.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:49.435 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:42:49.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:42:49.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:49.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:49.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:50.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:50.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:50.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:42:50.879 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:42:50.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:50.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:51.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:52.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:52.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:52.271 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:42:52.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:42:52.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:52.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:53.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:53.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:53.650 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:42:53.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:42:53.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:53.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:54.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:55.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:55.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:55.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:42:55.134 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:42:55.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:55.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:55.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:56.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:56.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:56.538 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:42:56.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:42:56.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:56.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:56.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:57.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:57.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:57.932 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:42:57.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:42:57.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:57.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:42:58.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:42:59.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:42:59.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:42:59.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:42:59.708 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:42:59.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:42:59.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:00.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:01.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:01.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:01.269 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:43:01.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:43:01.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:01.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:01.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:02.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:02.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:02.939 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:43:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:43:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:02.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:04.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:05.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:05.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:05.112 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:43:05.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:43:05.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:05.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:05.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:06.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:06.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:06.855 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:43:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:43:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:06.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:07.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:08.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:08.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:08.330 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:43:08.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:43:08.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:08.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:09.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:10.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:10.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:10.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:43:10.032 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:43:10.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:10.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:43:10.382 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/462348,v1:127.0.0.1:6803/462348] [v2:127.0.0.1:6804/462348,v1:127.0.0.1:6805/462348] exists,up 8edde77f-2569-4512-85cb-350ceb4db2aa 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:43:10.386 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:43:10.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:43:10.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=05ab6768-acc7-43f8-859c-7a4e2cad318f 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 05ab6768-acc7-43f8-859c-7a4e2cad318f' 2024-09-15T05:43:10.389 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 05ab6768-acc7-43f8-859c-7a4e2cad318f 2024-09-15T05:43:10.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:43:10.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDuc+ZmTp2xFxAAfzKjLxivFxc+D1P2GwluQw== 2024-09-15T05:43:10.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDuc+ZmTp2xFxAAfzKjLxivFxc+D1P2GwluQw=="}' 2024-09-15T05:43:10.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 05ab6768-acc7-43f8-859c-7a4e2cad318f -i td/osd-reuse-id/1/new.json 2024-09-15T05:43:10.904 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:43:10.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-15T05:43:10.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDuc+ZmTp2xFxAAfzKjLxivFxc+D1P2GwluQw== --osd-uuid 05ab6768-acc7-43f8-859c-7a4e2cad318f 2024-09-15T05:43:10.939 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.938+0000 7fd763213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:10.973 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.972+0000 7fd763213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:10.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.973+0000 7fd763213440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:10.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.973+0000 7fd763213440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:10.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.973+0000 7fd763213440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:10.974 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:10.973+0000 7fd763213440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:11.013 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:11.012+0000 7fd763213440 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-15T05:43:16.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-15T05:43:16.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:43:16.878 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:43:16.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:43:16.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:43:17.353 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:43:17.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:43:17.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:43:17.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:43:17.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:43:17.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:43:17.374 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:17.373+0000 7f7e8040c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:17.376 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:17.375+0000 7f7e8040c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:17.377 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:17.376+0000 7f7e8040c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:17.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:43:17.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:43:17.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:43:17.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:43:17.697 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:43:17.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:43:17.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:17.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:43:17.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:17.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:17.902 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:17.902+0000 7f7e8040c440 -1 Falling back to public interface 2024-09-15T05:43:18.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:19.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:19.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:19.059 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:43:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:43:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:19.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:19.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:20.387 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:20.385+0000 7f7e8040c440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:43:20.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:20.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:20.413 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:43:20.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:43:20.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:20.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:20.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:21.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:21.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:21.848 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:43:21.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:43:21.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:21.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:22.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:23.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:23.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:23.328 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:43:23.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:43:23.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:23.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:23.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:24.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:24.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:24.853 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:43:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:43:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:24.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:25.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:26.311 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:43:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:43:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:26.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:27.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:27.690 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:43:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:43:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:27.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:28.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:29.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:29.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:29.177 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:43:29.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:43:29.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:29.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:29.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:30.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:30.572 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:43:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:43:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:30.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:31.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:32.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:32.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:32.034 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:43:32.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:43:32.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:32.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:32.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:33.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:33.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:33.587 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:43:33.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:43:33.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:33.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:34.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:35.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:35.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:35.729 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:43:35.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:43:35.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:35.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:36.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:37.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:37.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:37.204 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:43:37.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:43:37.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:37.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:38.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:39.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:39.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:39.446 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:43:39.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:43:39.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:39.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:40.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:41.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:41.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:41.303 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:43:41.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:43:41.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:41.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:42.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:43.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:43.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:43.055 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:43:43.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:43:43.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:43.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:43.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:43:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:43:44.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:43:44.640 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:43:44.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:43:44.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:43:44.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:43:44.995 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/464104,v1:127.0.0.1:6811/464104] [v2:127.0.0.1:6812/464104,v1:127.0.0.1:6813/464104] exists,up 05ab6768-acc7-43f8-859c-7a4e2cad318f 2024-09-15T05:43:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:43:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:43:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:43:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-09-15T05:43:44.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-15T05:43:45.361 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' does not exist 2024-09-15T05:43:45.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-15T05:43:45.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-15T05:43:45.842 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'rbd' already exists 2024-09-15T05:43:45.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:43:46.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-15T05:43:48.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-09-15T05:43:48.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:43:48.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:43:48.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:43:48.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:43:48.246 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:43:48.246 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:43:48.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:43:48.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:43:48.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:43:48.340 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:43:48.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:43:48.686 INFO:tasks.workunit.client.0.smithi138.stderr:1' 2024-09-15T05:43:48.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:43:48.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:43:48.686 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:43:48.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836493 2024-09-15T05:43:48.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836493 2024-09-15T05:43:48.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493' 2024-09-15T05:43:48.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:43:48.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:43:48.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672967 2024-09-15T05:43:48.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672967 2024-09-15T05:43:48.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836493 1-42949672967' 2024-09-15T05:43:48.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:43:48.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836493 2024-09-15T05:43:48.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:43:48.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:43:48.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836493 2024-09-15T05:43:48.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:43:48.897 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836493 2024-09-15T05:43:48.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836493 2024-09-15T05:43:48.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-09-15T05:43:48.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:43:49.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836493 2024-09-15T05:43:49.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:43:50.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:43:50.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:43:50.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836491 -lt 21474836493 2024-09-15T05:43:50.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:43:51.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T05:43:51.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:43:51.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836494 -lt 21474836493 2024-09-15T05:43:51.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:43:51.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672967 2024-09-15T05:43:51.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:43:51.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:43:51.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672967 2024-09-15T05:43:51.962 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:43:51.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672967 2024-09-15T05:43:51.963 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672967 2024-09-15T05:43:51.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2024-09-15T05:43:51.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:43:52.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672968 -lt 42949672967 2024-09-15T05:43:52.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:43:52.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:43:52.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:43:52.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-15T05:43:52.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:43:52.896 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:43:53.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-15T05:43:53.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:43:53.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:43:53.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:43:53.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-09-15T05:43:53.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-09-15T05:43:54.255 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already out. 2024-09-15T05:43:54.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-09-15T05:43:54.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:43:54.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:43:54.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:43:54.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:43:54.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:43:54.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:43:54.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-09-15T05:43:54.975 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 is already down. 2024-09-15T05:43:54.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-09-15T05:43:55.397 INFO:tasks.workunit.client.0.smithi138.stderr:osd.1 does not exist 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:43:55.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:43:55.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:43:55.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:43:55.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:43:55.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:43:55.410 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:43:55.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:43:55.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:43:55.412 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:43:55.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:43:55.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:43:55.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:43:55.414 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:43:55.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:43:55.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:43:55.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-09-15T05:43:55.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:43:55.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:43:55.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:43:55.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.461257 2024-09-15T05:43:55.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:43:55.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:43:55.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:43:55.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:43:55.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-09-15T05:43:55.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:43:55.427 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 a2a7ecb1-9178-43c4-802a-a7ce028e7b43 2024-09-15T05:43:55.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2a7ecb1-9178-43c4-802a-a7ce028e7b43 2024-09-15T05:43:55.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a2a7ecb1-9178-43c4-802a-a7ce028e7b43' 2024-09-15T05:43:55.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:43:55.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbdOZmmabxGRAAUPg7IL2iLqjStSxuhAl+Gw== 2024-09-15T05:43:55.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbdOZmmabxGRAAUPg7IL2iLqjStSxuhAl+Gw=="}' 2024-09-15T05:43:55.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2a7ecb1-9178-43c4-802a-a7ce028e7b43 -i td/osd-reuse-id/1/new.json 2024-09-15T05:43:55.929 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:43:55.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-09-15T05:43:55.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbdOZmmabxGRAAUPg7IL2iLqjStSxuhAl+Gw== --osd-uuid a2a7ecb1-9178-43c4-802a-a7ce028e7b43 2024-09-15T05:43:55.964 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.963+0000 7f2688e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:55.966 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.965+0000 7f2688e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:55.967 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.965+0000 7f2688e12440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:43:55.967 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.966+0000 7f2688e12440 -1 asok(0x55ac0f2ce000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.461257/ceph-osd.1.asok': (2) No such file or directory 2024-09-15T05:43:55.967 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.966+0000 7f2688e12440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:55.967 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.966+0000 7f2688e12440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:55.967 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:55.966+0000 7f2688e12440 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-09-15T05:43:56.045 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:43:56.044+0000 7f2688e12440 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-09-15T05:44:01.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-09-15T05:44:01.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:44:01.642 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:44:01.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:44:01.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:44:02.124 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:44:02.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:44:02.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=cda220b6-e029-4ea4-8929-e52de7c3c351 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.461257/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:44:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:44:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:44:02.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:44:02.144 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:02.143+0000 7f3fb1527440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:02.146 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:02.145+0000 7f3fb1527440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:02.147 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:02.146+0000 7f3fb1527440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:02.147 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:02.146+0000 7f3fb1527440 -1 asok(0x5568ea7e4000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.461257/ceph-osd.1.asok': (2) No such file or directory 2024-09-15T05:44:02.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:44:02.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:44:02.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:44:02.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:44:02.514 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:44:02.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:44:02.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:02.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:44:02.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:02.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:02.673 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:02.672+0000 7f3fb1527440 -1 Falling back to public interface 2024-09-15T05:44:02.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:03.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:03.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:03.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:44:03.867 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:44:03.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:03.868 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:04.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:05.181 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:05.180+0000 7f3fb1527440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:44:05.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:05.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:05.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:44:05.218 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:44:05.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:05.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:05.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:06.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:06.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:06.776 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:44:06.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:44:06.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:06.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:07.120 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/466327,v1:127.0.0.1:6811/466327] [v2:127.0.0.1:6812/466327,v1:127.0.0.1:6813/466327] exists,up a2a7ecb1-9178-43c4-802a-a7ce028e7b43 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:44:07.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:44:07.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:44:07.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:44:07.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:44:07.227 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:44:07.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:44:07.229 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:44:07.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:44:07.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:44:07.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:44:07.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:44:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:44:07.231 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:44:07.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:44:07.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-15T05:44:07.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:44:07.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:07.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:44:07.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.461257 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/osd-reuse-id 0 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:44:07.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:44:07.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:44:07.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:44:07.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:44:07.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:44:07.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:44:07.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:44:07.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:44:07.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:44:07.246 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:44:07.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:44:07.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:44:07.248 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.248 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:44:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T05:44:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-09-15T05:44:07.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:44:07.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:07.250 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.461257 2024-09-15T05:44:07.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.461257 2024-09-15T05:44:07.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:44:07.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:44:07.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T05:44:07.252 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T05:44:07.252 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T05:44:07.316 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-09-15T05:44:07.317 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-09-15T05:44:07.378 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T05:44:07.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/pg-split-merge 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none ' 2024-09-15T05:44:07.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:44:07.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-09-15T05:44:07.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T05:44:07.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-09-15T05:44:07.387 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_import_after_merge_and_gap 2024-09-15T05:44:07.387 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_import_after_split' 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:44:07.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:44:07.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:44:07.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:44:07.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:44:07.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:44:07.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:44:07.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:44:07.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:44:07.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:44:07.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:44:07.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:44:07.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:44:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:44:07.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:44:07.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:44:07.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-15T05:44:07.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:44:07.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:07.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:07.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.466970 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-15T05:44:07.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-15T05:44:07.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:07.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-15T05:44:08.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:44:08.009 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:44:08.010 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get fsid 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:44:08.086 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:44:08.087 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get mon_host 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-15T05:44:08.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:44:08.985 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:44:08.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:44:09.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:44:09.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:44:09.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:44:09.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=69c8cf15-0614-45f4-bc1b-15ff9676e973 2024-09-15T05:44:09.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 69c8cf15-0614-45f4-bc1b-15ff9676e973' 2024-09-15T05:44:09.017 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 69c8cf15-0614-45f4-bc1b-15ff9676e973 2024-09-15T05:44:09.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:44:09.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApdOZm/lK5ARAAnCINppzDocqQjxfhLO9nOA== 2024-09-15T05:44:09.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApdOZm/lK5ARAAnCINppzDocqQjxfhLO9nOA=="}' 2024-09-15T05:44:09.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 69c8cf15-0614-45f4-bc1b-15ff9676e973 -i td/pg-split-merge/0/new.json 2024-09-15T05:44:09.331 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:44:09.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-15T05:44:09.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApdOZm/lK5ARAAnCINppzDocqQjxfhLO9nOA== --osd-uuid 69c8cf15-0614-45f4-bc1b-15ff9676e973 2024-09-15T05:44:09.367 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.365+0000 7ffa7728b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:09.376 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.374+0000 7ffa7728b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:09.377 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.375+0000 7ffa7728b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:09.378 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.376+0000 7ffa7728b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:44:09.378 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.376+0000 7ffa7728b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:44:09.378 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.376+0000 7ffa7728b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:44:09.416 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:09.415+0000 7ffa7728b440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-15T05:44:14.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-15T05:44:14.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:44:14.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:44:14.798 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:44:14.798 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:44:15.280 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:44:15.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:44:15.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:44:15.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:44:15.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:44:15.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:44:15.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:15.304+0000 7fa4e7a49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:15.306 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:15.306+0000 7fa4e7a49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:15.307 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:15.307+0000 7fa4e7a49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:15.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:44:15.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:44:15.632 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:44:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:15.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:15.834 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:15.833+0000 7fa4e7a49440 -1 Falling back to public interface 2024-09-15T05:44:15.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:16.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:16.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:16.991 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:44:16.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:44:16.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:16.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:17.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:18.320 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:18.319+0000 7fa4e7a49440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:44:18.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:18.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:18.348 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:44:18.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:44:18.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:18.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:18.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:19.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:19.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:19.707 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:44:19.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:44:19.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:19.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:20.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:21.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:21.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:21.144 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:44:21.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:44:21.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:21.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:21.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:22.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:22.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:22.535 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:44:22.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:44:22.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:22.536 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:22.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:23.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:23.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:23.998 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:44:23.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:44:23.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:23.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:24.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:25.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:25.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:25.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:44:25.394 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:44:25.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:25.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:25.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:26.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:26.800 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:26.800 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:44:26.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:44:26.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:26.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:27.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:28.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:28.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:28.306 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:44:28.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:44:28.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:28.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:28.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:29.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:29.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:29.703 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:44:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:44:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:29.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:30.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:31.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:31.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:31.515 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:44:31.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:44:31.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:31.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:32.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:33.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:33.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:33.271 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:44:33.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:44:33.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:33.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:34.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:35.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:35.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:35.118 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:44:35.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:44:35.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:35.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:35.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:36.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:36.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:36.932 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:44:36.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:44:36.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:36.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:37.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:38.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:38.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:38.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:44:38.668 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:44:38.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:38.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:39.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:40.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:40.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:40.474 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:44:40.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:44:40.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:40.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:40.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:41.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:41.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:41.907 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:44:41.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:44:41.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:41.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:44:42.304 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/468061,v1:127.0.0.1:6803/468061] [v2:127.0.0.1:6804/468061,v1:127.0.0.1:6805/468061] exists,up 69c8cf15-0614-45f4-bc1b-15ff9676e973 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-09-15T05:44:42.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:44:42.309 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:44:42.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f0c48d17-a20b-433a-a5f7-bee56add3a0e 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f0c48d17-a20b-433a-a5f7-bee56add3a0e' 2024-09-15T05:44:42.311 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 f0c48d17-a20b-433a-a5f7-bee56add3a0e 2024-09-15T05:44:42.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:44:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKdOZmoXsvExAAZvy6OlwNCXDzjHr+SnjxMQ== 2024-09-15T05:44:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKdOZmoXsvExAAZvy6OlwNCXDzjHr+SnjxMQ=="}' 2024-09-15T05:44:42.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f0c48d17-a20b-433a-a5f7-bee56add3a0e -i td/pg-split-merge/1/new.json 2024-09-15T05:44:42.845 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:44:42.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-09-15T05:44:42.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKdOZmoXsvExAAZvy6OlwNCXDzjHr+SnjxMQ== --osd-uuid f0c48d17-a20b-433a-a5f7-bee56add3a0e 2024-09-15T05:44:42.883 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.882+0000 7f788b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:42.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.884+0000 7f788b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.885+0000 7f788b808440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:42.886 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.885+0000 7f788b808440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-15T05:44:42.886 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.885+0000 7f788b808440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-15T05:44:42.886 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.885+0000 7f788b808440 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-09-15T05:44:42.961 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:42.960+0000 7f788b808440 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-09-15T05:44:48.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-09-15T05:44:48.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:44:48.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:44:48.753 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:44:48.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:44:49.231 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:44:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:44:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:44:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:44:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:44:49.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:44:49.256 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:49.255+0000 7f0f7831f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:49.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:49.257+0000 7f0f7831f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:49.259 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:49.258+0000 7f0f7831f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:44:49.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:44:49.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:44:49.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:44:49.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:44:49.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:44:49.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:49.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:44:49.597 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:44:49.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:49.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:49.780 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:49.779+0000 7f0f7831f440 -1 Falling back to public interface 2024-09-15T05:44:49.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:50.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:50.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:50.961 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:44:50.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:44:50.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:50.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:51.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:52.294 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:44:52.293+0000 7f0f7831f440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:44:52.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:52.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:52.340 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:44:52.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:44:52.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:52.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:52.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:53.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:53.965 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:44:53.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:44:53.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:53.966 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:54.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:55.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:55.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:55.356 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:44:55.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:44:55.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:55.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:55.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:56.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:56.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:56.737 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:44:56.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:44:56.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:56.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:57.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:58.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:58.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:58.168 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:44:58.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:44:58.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:58.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:44:58.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:44:59.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:44:59.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:44:59.583 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:44:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:44:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:44:59.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:00.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:01.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:01.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:01.084 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:45:01.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:45:01.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:01.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:01.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:02.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:02.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:02.539 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:45:02.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:45:02.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:02.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:02.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:03.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:03.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:03.914 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:45:03.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:45:03.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:03.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:04.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:05.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:05.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:05.613 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:45:05.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:45:05.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:05.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:06.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:07.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:07.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:07.549 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:45:07.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:45:07.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:07.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:08.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:09.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:09.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:09.350 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:45:09.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:45:09.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:09.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:10.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:11.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:11.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:11.215 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:45:11.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:45:11.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:11.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:12.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:13.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:13.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:13.234 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:45:13.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:45:13.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:13.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:13.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:14.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:14.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:14.753 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:45:14.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:45:14.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:14.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:15.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:16.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:16.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:16.783 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:45:16.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:45:16.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:16.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:45:17.185 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/469815,v1:127.0.0.1:6811/469815] [v2:127.0.0.1:6812/469815,v1:127.0.0.1:6813/469815] exists,up f0c48d17-a20b-433a-a5f7-bee56add3a0e 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-09-15T05:45:17.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:45:17.190 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:45:17.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-09-15T05:45:17.192 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:45:17.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58c432e8-57a3-4f8d-8acc-d624511695ce 2024-09-15T05:45:17.192 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 58c432e8-57a3-4f8d-8acc-d624511695ce 2024-09-15T05:45:17.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 58c432e8-57a3-4f8d-8acc-d624511695ce' 2024-09-15T05:45:17.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:45:17.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBtdOZmzK0oDBAA3l0zgMbZcqfCwszVIaF44Q== 2024-09-15T05:45:17.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBtdOZmzK0oDBAA3l0zgMbZcqfCwszVIaF44Q=="}' 2024-09-15T05:45:17.205 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58c432e8-57a3-4f8d-8acc-d624511695ce -i td/pg-split-merge/2/new.json 2024-09-15T05:45:17.676 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:45:17.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-09-15T05:45:17.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBtdOZmzK0oDBAA3l0zgMbZcqfCwszVIaF44Q== --osd-uuid 58c432e8-57a3-4f8d-8acc-d624511695ce 2024-09-15T05:45:17.714 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.713+0000 7fa09ae1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:17.716 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.715+0000 7fa09ae1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:17.717 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.716+0000 7fa09ae1a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:17.717 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.716+0000 7fa09ae1a440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-15T05:45:17.718 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.716+0000 7fa09ae1a440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-15T05:45:17.718 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.716+0000 7fa09ae1a440 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-09-15T05:45:17.767 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:17.766+0000 7fa09ae1a440 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-09-15T05:45:23.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-09-15T05:45:23.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:45:23.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:45:23.566 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:45:23.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:45:24.056 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:45:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:45:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:45:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:45:24.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:45:24.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:45:24.080 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:24.079+0000 7f7898726440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:24.109 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:24.108+0000 7f7898726440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:24.110 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:24.109+0000 7f7898726440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:45:24.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:45:24.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:45:24.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:45:24.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:45:24.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:45:24.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:24.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:45:24.422 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:45:24.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:24.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:24.628 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:24.627+0000 7f7898726440 -1 Falling back to public interface 2024-09-15T05:45:24.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:25.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:25.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:25.806 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:45:25.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:45:25.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:25.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:26.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:27.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:27.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:27.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:45:27.167 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:45:27.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:27.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:27.279 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:45:27.278+0000 7f7898726440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:45:27.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:28.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:28.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:28.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:45:28.597 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:45:28.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:28.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:29.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:30.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:30.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:30.063 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:45:30.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:45:30.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:30.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:30.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:45:31.483 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:45:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:31.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:32.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:32.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:32.856 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:45:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:45:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:33.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:34.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:34.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:34.338 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:45:34.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:45:34.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:34.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:34.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:35.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:35.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:35.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:45:35.708 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:45:35.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:35.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:36.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:37.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:37.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:37.174 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:45:37.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:45:37.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:37.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:37.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:38.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:38.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:38.657 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:45:38.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:45:38.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:38.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:40.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:40.111 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:40.112 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:45:40.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:45:40.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:40.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:40.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:41.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:41.839 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:41.840 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:45:41.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:45:41.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:41.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:42.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:43.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:43.733 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:45:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:45:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:43.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:44.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:45.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:45.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:45.787 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:45:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:45:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:45.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:46.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:47.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:47.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:47.556 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:45:47.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:45:47.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:47.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:48.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:49.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:49.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:49.453 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:45:49.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:45:49.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:49.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:50.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:45:51.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:45:51.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:45:51.281 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:45:51.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:45:51.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:45:51.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:45:51.642 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/471570,v1:127.0.0.1:6819/471570] [v2:127.0.0.1:6820/471570,v1:127.0.0.1:6821/471570] exists,up 58c432e8-57a3-4f8d-8acc-d624511695ce 2024-09-15T05:45:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:45:51.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:45:51.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:45:51.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-09-15T05:45:52.350 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:45:52.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-09-15T05:45:53.440 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 pgp_num to 1 2024-09-15T05:45:53.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-09-15T05:45:53.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:45:53.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:45:53.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:45:53.452 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:45:53.452 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:45:53.453 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:45:53.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:45:53.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:45:53.453 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:45:53.538 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:45:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:45:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:45:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:45:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:45:54.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:54.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:45:54.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836503 2024-09-15T05:45:54.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836503 2024-09-15T05:45:54.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503' 2024-09-15T05:45:54.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:54.152 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:45:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672972 2024-09-15T05:45:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672972 2024-09-15T05:45:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672972' 2024-09-15T05:45:54.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:54.320 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:45:54.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509446 2024-09-15T05:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509446 2024-09-15T05:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836503 1-42949672972 2-64424509446' 2024-09-15T05:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836503 2024-09-15T05:45:54.428 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:45:54.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:45:54.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836503 2024-09-15T05:45:54.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:45:54.430 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836503 2024-09-15T05:45:54.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836503 2024-09-15T05:45:54.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836503' 2024-09-15T05:45:54.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:45:54.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836503 -lt 21474836503 2024-09-15T05:45:54.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:45:54.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672972 2024-09-15T05:45:54.791 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:45:54.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:45:54.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672972 2024-09-15T05:45:54.792 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:45:54.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672972 2024-09-15T05:45:54.793 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672972 2024-09-15T05:45:54.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672972' 2024-09-15T05:45:54.793 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:45:55.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672972 -lt 42949672972 2024-09-15T05:45:55.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:45:55.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509446 2024-09-15T05:45:55.268 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:45:55.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:45:55.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509446 2024-09-15T05:45:55.269 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:45:55.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509446 2024-09-15T05:45:55.271 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509446 2024-09-15T05:45:55.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2024-09-15T05:45:55.271 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:45:55.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509446 -lt 64424509446 2024-09-15T05:45:55.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:45:55.731 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:45:55.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:45:56.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:45:56.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:45:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:45:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:45:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:45:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:45:56.208 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:45:56.209 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:45:56.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:45:56.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:45:56.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:45:56.548 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:45:57.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:45:57.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:45:57.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:45:57.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-09-15T05:45:57.166 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-09-15T05:45:57.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-09-15T05:45:57.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-09-15T05:45:57.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-09-15T05:45:57.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-09-15T05:45:57.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-09-15T05:45:58.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-09-15T05:45:58.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-09-15T05:45:58.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-09-15T05:45:58.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-09-15T05:45:58.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-09-15T05:45:58.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:45:58.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:45:58.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:45:58.633 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:45:58.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:45:58.634 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:45:58.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:45:58.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:45:58.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:45:58.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:45:58.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:45:58.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:45:58.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:45:58.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:45:58.720 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:59.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:45:59.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836506 2024-09-15T05:45:59.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836506 2024-09-15T05:45:59.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506' 2024-09-15T05:45:59.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:59.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:45:59.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672975 2024-09-15T05:45:59.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672975 2024-09-15T05:45:59.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672975' 2024-09-15T05:45:59.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:45:59.283 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:45:59.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509449 2024-09-15T05:45:59.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509449 2024-09-15T05:45:59.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836506 1-42949672975 2-64424509449' 2024-09-15T05:45:59.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:45:59.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836506 2024-09-15T05:45:59.391 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:45:59.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:45:59.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836506 2024-09-15T05:45:59.392 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:45:59.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836506 2024-09-15T05:45:59.393 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836506 2024-09-15T05:45:59.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-09-15T05:45:59.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:45:59.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836504 -lt 21474836506 2024-09-15T05:45:59.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:46:00.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:46:00.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:01.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836506 -lt 21474836506 2024-09-15T05:46:01.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:01.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672975 2024-09-15T05:46:01.096 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:01.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:46:01.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672975 2024-09-15T05:46:01.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:01.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672975 2024-09-15T05:46:01.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-09-15T05:46:01.099 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672975 2024-09-15T05:46:01.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:46:01.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672975 -lt 42949672975 2024-09-15T05:46:01.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:01.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509449 2024-09-15T05:46:01.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:01.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:46:01.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509449 2024-09-15T05:46:01.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:01.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509449 2024-09-15T05:46:01.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-15T05:46:01.458 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509449 2024-09-15T05:46:01.459 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:46:01.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-15T05:46:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:46:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:01.806 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:02.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:46:02.278 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:46:02.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:46:02.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:46:02.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:02.621 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:46:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:46:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:46:03.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-09-15T05:46:03.211 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T05:46:03.303 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "" 2024-09-15T05:46:03.303 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T05:46:03.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-09-15T05:46:04.161 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 2 2024-09-15T05:46:04.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-09-15T05:46:04.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:46:04.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:46:04.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:46:04.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:46:04.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:04.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:46:04.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T05:46:04.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T05:46:04.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T05:46:04.849 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:04.849 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:46:04.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672978 2024-09-15T05:46:04.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672978 2024-09-15T05:46:04.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672978' 2024-09-15T05:46:04.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:04.992 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:46:05.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509452 2024-09-15T05:46:05.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509452 2024-09-15T05:46:05.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-42949672978 2-64424509452' 2024-09-15T05:46:05.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:05.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T05:46:05.097 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:05.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:46:05.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T05:46:05.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:05.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T05:46:05.099 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836509 2024-09-15T05:46:05.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T05:46:05.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:05.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-15T05:46:05.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:46:06.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:46:06.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:06.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836509 -lt 21474836509 2024-09-15T05:46:06.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:06.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672978 2024-09-15T05:46:06.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:06.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:46:06.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672978 2024-09-15T05:46:06.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:06.816 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672978 2024-09-15T05:46:06.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672978 2024-09-15T05:46:06.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-09-15T05:46:06.816 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:46:07.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672978 -lt 42949672978 2024-09-15T05:46:07.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:07.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509452 2024-09-15T05:46:07.190 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:07.191 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:46:07.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509452 2024-09-15T05:46:07.191 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:07.192 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509452 2024-09-15T05:46:07.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509452 2024-09-15T05:46:07.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-15T05:46:07.193 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:46:07.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509452 -lt 64424509452 2024-09-15T05:46:07.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:46:07.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:07.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:46:08.022 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:46:08.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:46:08.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:46:08.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:08.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:46:08.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:46:09.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:46:09.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-09-15T05:46:10.411 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:46:10.411 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:46:10.411 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-15T05:46:10.496 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:46:10.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-09-15T05:46:10.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:46:10.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:46:10.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:46:10.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:46:10.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:46:10.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:46:10.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:46:10.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:46:10.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-09-15T05:46:10.931 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:46:10.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:46:10.955 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:10.954+0000 7fb6b4921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:10.957 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:10.956+0000 7fb6b4921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:10.958 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:10.957+0000 7fb6b4921440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:11.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T05:46:11.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:46:11.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:46:11.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:46:11.484 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:11.483+0000 7fb6b4921440 -1 Falling back to public interface 2024-09-15T05:46:11.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:12.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:12.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:12.654 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:46:12.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:46:12.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:12.655 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:46:13.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:14.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:14.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:14.020 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:46:14.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:46:14.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:14.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:46:14.147 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:14.146+0000 7fb6b4921440 -1 osd.2 25 log_to_monitors true 2024-09-15T05:46:14.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:15.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:15.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:15.428 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:46:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:46:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:46:15.830 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 28 up_thru 17 down_at 26 last_clean_interval [15,25) [v2:127.0.0.1:6818/474175,v1:127.0.0.1:6819/474175] [v2:127.0.0.1:6820/474175,v1:127.0.0.1:6821/474175] exists,up 58c432e8-57a3-4f8d-8acc-d624511695ce 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:46:15.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:46:15.833 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:46:15.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:46:15.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:46:15.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:46:15.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:46:15.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:46:15.918 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:16.267 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:46:16.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836513 2024-09-15T05:46:16.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836513 2024-09-15T05:46:16.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513' 2024-09-15T05:46:16.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:16.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:46:16.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672982 2024-09-15T05:46:16.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672982 2024-09-15T05:46:16.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672982' 2024-09-15T05:46:16.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:16.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:46:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084291 2024-09-15T05:46:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084291 2024-09-15T05:46:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836513 1-42949672982 2-120259084291' 2024-09-15T05:46:16.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:16.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836513 2024-09-15T05:46:16.589 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:16.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:46:16.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836513 2024-09-15T05:46:16.590 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:16.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836513 2024-09-15T05:46:16.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-09-15T05:46:16.591 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836513 2024-09-15T05:46:16.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836513 -lt 21474836513 2024-09-15T05:46:16.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:16.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672982 2024-09-15T05:46:16.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:16.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:46:16.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672982 2024-09-15T05:46:16.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:16.939 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672982 2024-09-15T05:46:16.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672982 2024-09-15T05:46:16.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-09-15T05:46:16.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:46:17.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672982 -lt 42949672982 2024-09-15T05:46:17.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:17.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-120259084291 2024-09-15T05:46:17.302 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:17.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:46:17.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-120259084291 2024-09-15T05:46:17.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:17.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084291 2024-09-15T05:46:17.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 120259084291' 2024-09-15T05:46:17.305 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 120259084291 2024-09-15T05:46:17.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:46:17.656 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 120259084291 2024-09-15T05:46:17.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:46:18.657 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:46:18.658 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:46:19.017 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084291 -lt 120259084291 2024-09-15T05:46:19.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:46:19.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:19.018 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:19.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:46:19.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:46:19.487 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:46:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:46:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:46:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:46:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:46:19.488 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:46:19.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:46:19.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:46:19.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:19.831 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:20.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:46:20.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:46:20.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:46:20.342 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-09-15T05:46:20.441 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T05:46:20.441 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "" 2024-09-15T05:46:20.441 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T05:46:20.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-09-15T05:46:21.258 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already out. osd.1 is already out. 2024-09-15T05:46:21.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-09-15T05:46:22.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-09-15T05:46:27.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-09-15T05:46:27.134 INFO:tasks.workunit.client.0.smithi138.stdout:{ 2024-09-15T05:46:27.134 INFO:tasks.workunit.client.0.smithi138.stdout: "success": "" 2024-09-15T05:46:27.134 INFO:tasks.workunit.client.0.smithi138.stdout:} 2024-09-15T05:46:27.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-09-15T05:46:27.764 INFO:tasks.workunit.client.0.smithi138.stderr:osd.0 is already in. osd.1 is already in. 2024-09-15T05:46:27.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-09-15T05:46:28.828 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T05:46:28.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-09-15T05:46:28.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:46:28.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:46:28.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:46:28.840 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:46:28.841 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:46:28.841 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:46:28.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:46:28.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:46:28.841 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:46:28.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:46:29.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:46:29.442 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:46:29.442 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T05:46:29.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:46:29.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:29.442 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:46:29.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836518 2024-09-15T05:46:29.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836518 2024-09-15T05:46:29.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518' 2024-09-15T05:46:29.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:29.598 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:46:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672987 2024-09-15T05:46:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672987 2024-09-15T05:46:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672987' 2024-09-15T05:46:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:46:29.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:46:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=120259084296 2024-09-15T05:46:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 120259084296 2024-09-15T05:46:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836518 1-42949672987 2-120259084296' 2024-09-15T05:46:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:29.933 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836518 2024-09-15T05:46:29.934 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:29.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:46:29.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836518 2024-09-15T05:46:29.935 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:29.935 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836518 2024-09-15T05:46:29.936 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836518 2024-09-15T05:46:29.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2024-09-15T05:46:29.936 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:30.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836516 -lt 21474836518 2024-09-15T05:46:30.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:46:31.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:46:31.354 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:46:31.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836518 -lt 21474836518 2024-09-15T05:46:31.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:31.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672987 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672987 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672987 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672987' 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672987 2024-09-15T05:46:31.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:46:32.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672987 -lt 42949672987 2024-09-15T05:46:32.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:46:32.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-120259084296 2024-09-15T05:46:32.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:46:32.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:46:32.170 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-120259084296 2024-09-15T05:46:32.170 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:46:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=120259084296 2024-09-15T05:46:32.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 120259084296' 2024-09-15T05:46:32.171 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 120259084296 2024-09-15T05:46:32.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:46:32.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 120259084296 -lt 120259084296 2024-09-15T05:46:32.534 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:46:32.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:32.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:46:33.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:46:33.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:46:33.007 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:46:33.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:46:33.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:46:33.345 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:46:33.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:46:33.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:46:33.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:46:33.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:46:33.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-09-15T05:46:34.149 INFO:tasks.workunit.client.0.smithi138.stderr:instructing pg 1.0 on osd.1 to scrub 2024-09-15T05:46:34.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-09-15T05:46:44.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-09-15T05:46:44.648 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:27.521235+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e35: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:28.090736+0000 mon.a (mon.0) 394 : cluster [DBG] osdmap e36: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:28.442585+0000 mon.a (mon.0) 396 : cluster [DBG] osdmap e37: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:28.828467+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:28.503621+0000 mgr.x (mgr.4100) 68 : cluster [DBG] pgmap v96: 2 pgs: 2 unknown; 88 KiB data, 291 MiB used, 100 GiB / 100 GiB avail 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:29.214363+0000 mon.a (mon.0) 400 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:29.499109+0000 mon.a (mon.0) 403 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:29.951799+0000 mon.a (mon.0) 404 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:30.000740+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (23'7,23'7] MIN to 23'7 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:30.773008+0000 mon.a (mon.0) 407 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-09-15T05:46:44.649 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:30.504060+0000 mgr.x (mgr.4100) 69 : cluster [DBG] pgmap v101: 1 pgs: 1 remapped+peering; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 25 KiB/s wr, 1 op/s 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:31.166927+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:32.504362+0000 mgr.x (mgr.4100) 70 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 16 KiB/s wr, 1 op/s; 15 KiB/s, 1 objects/s recovering 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:34.504638+0000 mgr.x (mgr.4100) 75 : cluster [DBG] pgmap v105: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 10 KiB/s wr, 0 op/s; 10 KiB/s, 0 objects/s recovering 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:35.792378+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:35.794556+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:36.505086+0000 mgr.x (mgr.4100) 76 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 7.7 KiB/s, 0 objects/s recovering 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:38.505505+0000 mgr.x (mgr.4100) 77 : cluster [DBG] pgmap v107: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 6.3 KiB/s, 0 objects/s recovering 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:40.505917+0000 mgr.x (mgr.4100) 78 : cluster [DBG] pgmap v108: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 5.2 KiB/s, 0 objects/s recovering 2024-09-15T05:46:44.650 INFO:tasks.workunit.client.0.smithi138.stdout:2024-09-15T05:46:42.506310+0000 mgr.x (mgr.4100) 79 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 138 KiB data, 873 MiB used, 299 GiB / 300 GiB avail; 4.4 KiB/s, 0 objects/s recovering 2024-09-15T05:46:44.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-09-15T05:46:45.031 INFO:tasks.workunit.client.0.smithi138.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-09-15T05:46:45.032 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 11 0 0 0 141625 0 0 0 11 active+clean 9s 23'7 43:97 [1,0,2]p1 [1,0,2]p1 2024-09-15T05:46:35.794561+0000 2024-09-15T05:45:51.960085+0000 1 periodic scrub scheduled @ 2024-09-16T08:30:46.411705+0000 2024-09-15T05:46:45.032 INFO:tasks.workunit.client.0.smithi138.stdout: 2024-09-15T05:46:45.032 INFO:tasks.workunit.client.0.smithi138.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-09-15T05:46:45.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-09-15T05:46:45.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-09-15T05:46:45.386 INFO:tasks.workunit.client.0.smithi138.stdout:1.0 11 0 0 0 141625 0 0 0 11 active+clean 9s 23'7 43:97 [1,0,2]p1 [1,0,2]p1 2024-09-15T05:46:35.794561+0000 2024-09-15T05:45:51.960085+0000 1 periodic scrub scheduled @ 2024-09-16T08:30:46.411705+0000 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:46:45.387 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:46:45.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:46:45.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:46:45.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:46:45.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:46:45.494 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:46:45.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:46:45.495 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:46:45.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:46:45.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:46:45.497 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:46:45.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:46:45.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:46:45.498 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:46:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:46:45.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:46:45.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:46:45.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:46:45.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:46:45.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:45.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:45.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-15T05:46:45.518 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:46:45.519 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:46:45.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:46:45.520 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:46:45.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:46:45.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:46:45.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:46:45.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:46:45.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:46:45.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:46:45.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:46:45.523 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:46:45.523 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:46:45.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:46:45.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:46:45.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:46:45.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:46:45.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:45.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:45.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:46:45.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:46:45.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:46:45.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-15T05:46:45.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:46:45.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:45.529 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:45.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.466970 2024-09-15T05:46:45.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-15T05:46:45.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:46:45.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:46:45.829 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:45.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:46:46.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:46:46.078 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:46:46.079 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:46:46.080 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get fsid 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:46:46.153 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:46.154 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:46:46.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:46:46.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get mon_host 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-15T05:46:46.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:47.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:46:47.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:46:47.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-09-15T05:46:47.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-15T05:46:47.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:46:47.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:46:47.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:46:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:46:47.103 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:46:47.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:46:47.103 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:46:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 be0cdffb-84ca-4a98-9332-b521f602a977' 2024-09-15T05:46:47.104 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:46:47.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHdOZmW/3XBhAAcRTt/oUMYvZ5QsM+n6QAxA== 2024-09-15T05:46:47.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHdOZmW/3XBhAAcRTt/oUMYvZ5QsM+n6QAxA=="}' 2024-09-15T05:46:47.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new be0cdffb-84ca-4a98-9332-b521f602a977 -i td/pg-split-merge/0/new.json 2024-09-15T05:46:47.434 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:46:47.444 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-15T05:46:47.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHdOZmW/3XBhAAcRTt/oUMYvZ5QsM+n6QAxA== --osd-uuid be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:46:47.470 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.469+0000 7f64bde3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:47.479 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.479+0000 7f64bde3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:47.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.480+0000 7f64bde3b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:47.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.480+0000 7f64bde3b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:46:47.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.480+0000 7f64bde3b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:46:47.481 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.480+0000 7f64bde3b440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:46:47.520 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:47.519+0000 7f64bde3b440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-15T05:46:52.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-15T05:46:52.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:46:52.862 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:46:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:46:52.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:46:53.350 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:46:53.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:46:53.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:46:53.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:46:53.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:46:53.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:46:53.375 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:53.374+0000 7f8158e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:53.377 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:53.376+0000 7f8158e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:53.378 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:53.377+0000 7f8158e0c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:53.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:46:53.702 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:46:53.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:53.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:46:53.895 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:53.894+0000 7f8158e0c440 -1 Falling back to public interface 2024-09-15T05:46:54.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:55.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:55.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:55.062 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:46:55.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:46:55.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:55.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:46:55.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:56.367 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:46:56.366+0000 7f8158e0c440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:46:56.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:56.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:56.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:46:56.422 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:46:56.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:56.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:46:56.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:57.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:57.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:57.971 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:46:57.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:46:57.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:57.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:46:58.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:46:59.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:46:59.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:46:59.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:46:59.337 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:46:59.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:46:59.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:46:59.697 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:00.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:00.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:00.698 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:47:00.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:47:00.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:00.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:01.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:02.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:02.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:02.231 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:47:02.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:47:02.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:02.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:02.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:03.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:03.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:03.620 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:47:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:47:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:03.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:04.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:05.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:05.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:05.046 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:47:05.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:47:05.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:05.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:05.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:06.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:06.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:06.425 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:47:06.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:47:06.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:06.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:06.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:07.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:07.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:07.807 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:47:07.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:47:07.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:07.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:09.571 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:47:09.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:47:09.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:09.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:10.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:11.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:11.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:11.045 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:47:11.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:47:11.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:11.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:11.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:12.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:12.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:12.774 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:47:12.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:47:12.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:12.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:13.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:14.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:14.241 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:47:14.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:47:14.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:14.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:14.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:15.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:15.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:15.915 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:47:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:47:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:16.658 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:17.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:17.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:17.660 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:47:17.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:47:17.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:17.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:18.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:19.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:19.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:19.461 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:47:19.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:47:19.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:19.462 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:19.882 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/476751,v1:127.0.0.1:6803/476751] [v2:127.0.0.1:6804/476751,v1:127.0.0.1:6805/476751] exists,up be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:47:19.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:47:19.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:47:19.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:47:19.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-09-15T05:47:20.415 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:47:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-15T05:47:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:47:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:47:20.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:47:20.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:47:20.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:47:20.427 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:47:20.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:47:20.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:47:20.427 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:47:20.512 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:47:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:47:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:47:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:47:20.903 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836487 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836487 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836487' 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836487 2024-09-15T05:47:21.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:47:21.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:47:21.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836487 2024-09-15T05:47:21.048 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:47:21.049 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836487 2024-09-15T05:47:21.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836487 2024-09-15T05:47:21.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-15T05:47:21.050 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:47:21.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-15T05:47:21.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:47:22.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:47:22.554 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:47:22.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836487 -lt 21474836487 2024-09-15T05:47:22.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:47:22.938 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:47:22.939 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:47:23.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:47:23.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:47:23.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:47:23.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:47:23.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:47:23.795 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:47:24.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:47:24.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:47:24.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:47:24.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-15T05:47:24.287 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T05:47:24.288 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-15T05:47:24.288 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_477915 2024-09-15T05:47:24.288 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T05:47:24.288 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T05:47:25.288 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 128 112 0.109362 0.109375 0.13474 0.128868 2024-09-15T05:47:26.288 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 272 256 0.124985 0.140625 0.117408 0.122486 2024-09-15T05:47:27.288 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 402 386 0.125637 0.126953 0.117512 0.118762 2024-09-15T05:47:28.288 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 3.09157 2024-09-15T05:47:28.288 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 402 2024-09-15T05:47:28.288 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 1024 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 1024 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.126984 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.0156656 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.140625 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.109375 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 130 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 16.0416 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 144 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 112 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.120707 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0270047 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.184334 2024-09-15T05:47:28.289 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0819555 2024-09-15T05:47:28.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-15T05:47:28.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:47:28.358 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:47:28.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:47:28.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:47:28.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:47:28.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:47:28.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-15T05:47:29.921 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.1 info 1.1( v 9'194 (0'0,9'194] local-lis/les=7/8 n=194 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:47:29.923 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81cead58:::benchmark_data_smithi138_477915_object95:head# 2024-09-15T05:47:29.923 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81e31d0f:::benchmark_data_smithi138_477915_object203:head# 2024-09-15T05:47:29.925 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:829c9f42:::benchmark_data_smithi138_477915_object297:head# 2024-09-15T05:47:29.927 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8326356e:::benchmark_data_smithi138_477915_object272:head# 2024-09-15T05:47:29.928 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:84ba03a2:::benchmark_data_smithi138_477915_object268:head# 2024-09-15T05:47:29.929 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:84bb0071:::benchmark_data_smithi138_477915_object169:head# 2024-09-15T05:47:29.929 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8531dfd7:::benchmark_data_smithi138_477915_object311:head# 2024-09-15T05:47:29.930 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:853bf743:::benchmark_data_smithi138_477915_object257:head# 2024-09-15T05:47:29.930 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:86e0b007:::benchmark_data_smithi138_477915_object74:head# 2024-09-15T05:47:29.932 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:86e93436:::benchmark_data_smithi138_477915_object319:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8709604b:::benchmark_data_smithi138_477915_object347:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:87d6852e:::benchmark_data_smithi138_477915_object231:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:88c42840:::benchmark_data_smithi138_477915_object140:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:89a81864:::benchmark_data_smithi138_477915_object264:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8a16f6c1:::benchmark_data_smithi138_477915_object93:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8aa67a0a:::benchmark_data_smithi138_477915_object235:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8aedb92d:::benchmark_data_smithi138_477915_object365:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8be95c11:::benchmark_data_smithi138_477915_object323:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8c3675a7:::benchmark_data_smithi138_477915_object303:head# 2024-09-15T05:47:29.933 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d8d0588:::benchmark_data_smithi138_477915_object28:head# 2024-09-15T05:47:29.938 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f046a5e:::benchmark_data_smithi138_477915_object109:head# 2024-09-15T05:47:29.940 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f04eb67:::benchmark_data_smithi138_477915_object160:head# 2024-09-15T05:47:29.941 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:900c1e9a:::benchmark_data_smithi138_477915_object168:head# 2024-09-15T05:47:29.942 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9033da55:::benchmark_data_smithi138_477915_object23:head# 2024-09-15T05:47:29.942 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90dfb114:::benchmark_data_smithi138_477915_object233:head# 2024-09-15T05:47:29.943 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90f41851:::benchmark_data_smithi138_477915_object256:head# 2024-09-15T05:47:29.944 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91c6592f:::benchmark_data_smithi138_477915_object36:head# 2024-09-15T05:47:29.944 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91f02d94:::benchmark_data_smithi138_477915_object249:head# 2024-09-15T05:47:29.944 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:92efb357:::benchmark_data_smithi138_477915_object242:head# 2024-09-15T05:47:29.944 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9432f393:::benchmark_data_smithi138_477915_object16:head# 2024-09-15T05:47:29.945 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:971a84a9:::benchmark_data_smithi138_477915_object149:head# 2024-09-15T05:47:29.945 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:971af863:::benchmark_data_smithi138_477915_object151:head# 2024-09-15T05:47:29.945 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a712198:::benchmark_data_smithi138_477915_object218:head# 2024-09-15T05:47:29.945 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9df1f34f:::benchmark_data_smithi138_477915_object356:head# 2024-09-15T05:47:29.946 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f9adf98:::benchmark_data_smithi138_477915_object200:head# 2024-09-15T05:47:29.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a080aed2:::benchmark_data_smithi138_477915_object104:head# 2024-09-15T05:47:29.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0aeaa72:::benchmark_data_smithi138_477915_object254:head# 2024-09-15T05:47:29.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0d22f50:::benchmark_data_smithi138_477915_object71:head# 2024-09-15T05:47:29.956 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a10a53f5:::benchmark_data_smithi138_477915_object298:head# 2024-09-15T05:47:29.957 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a10dc166:::benchmark_data_smithi138_477915_object117:head# 2024-09-15T05:47:29.957 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a1460743:::benchmark_data_smithi138_477915_object334:head# 2024-09-15T05:47:29.958 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a14ca631:::benchmark_data_smithi138_477915_object105:head# 2024-09-15T05:47:29.958 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a1c26988:::benchmark_data_smithi138_477915_object351:head# 2024-09-15T05:47:29.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a2e8a8e9:::benchmark_data_smithi138_477915_object184:head# 2024-09-15T05:47:29.960 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a5019f41:::benchmark_data_smithi138_477915_object302:head# 2024-09-15T05:47:29.961 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a574d19f:::benchmark_data_smithi138_477915_object27:head# 2024-09-15T05:47:29.961 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a5add657:::benchmark_data_smithi138_477915_object96:head# 2024-09-15T05:47:29.962 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a65ba2e1:::benchmark_data_smithi138_477915_object348:head# 2024-09-15T05:47:29.962 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a76a8553:::benchmark_data_smithi138_477915_object384:head# 2024-09-15T05:47:29.963 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a84031a3:::benchmark_data_smithi138_477915_object156:head# 2024-09-15T05:47:29.963 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a89e1157:::benchmark_data_smithi138_477915_object66:head# 2024-09-15T05:47:29.964 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a8ec4680:::benchmark_data_smithi138_477915_object280:head# 2024-09-15T05:47:29.964 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a91194b3:::benchmark_data_smithi138_477915_object68:head# 2024-09-15T05:47:29.965 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aae671fe:::benchmark_data_smithi138_477915_object173:head# 2024-09-15T05:47:29.965 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:abf706c3:::benchmark_data_smithi138_477915_object121:head# 2024-09-15T05:47:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ad8a20c0:::benchmark_data_smithi138_477915_object375:head# 2024-09-15T05:47:29.966 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ade77a16:::benchmark_data_smithi138_477915_object88:head# 2024-09-15T05:47:29.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ae6263f9:::benchmark_data_smithi138_477915_object31:head# 2024-09-15T05:47:29.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b05f7e87:::benchmark_data_smithi138_477915_object316:head# 2024-09-15T05:47:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b0627822:::benchmark_data_smithi138_477915_object255:head# 2024-09-15T05:47:29.968 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b08f0d28:::benchmark_data_smithi138_477915_object229:head# 2024-09-15T05:47:29.969 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b1344ffa:::benchmark_data_smithi138_477915_object33:head# 2024-09-15T05:47:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b142aea1:::benchmark_data_smithi138_477915_object64:head# 2024-09-15T05:47:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b2b53955:::benchmark_data_smithi138_477915_object199:head# 2024-09-15T05:47:29.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b2b94858:::benchmark_data_smithi138_477915_object345:head# 2024-09-15T05:47:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b30ed2d9:::benchmark_data_smithi138_477915_object224:head# 2024-09-15T05:47:29.971 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b36e2fb6:::benchmark_data_smithi138_477915_object56:head# 2024-09-15T05:47:29.972 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b388f84c:::benchmark_data_smithi138_477915_object270:head# 2024-09-15T05:47:29.972 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b3bb6a6a:::benchmark_data_smithi138_477915_object251:head# 2024-09-15T05:47:29.972 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b3ceb653:::benchmark_data_smithi138_477915_object130:head# 2024-09-15T05:47:29.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b413d550:::benchmark_data_smithi138_477915_object76:head# 2024-09-15T05:47:29.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b49d13a3:::benchmark_data_smithi138_477915_object296:head# 2024-09-15T05:47:29.974 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b4dcf7e3:::benchmark_data_smithi138_477915_object161:head# 2024-09-15T05:47:29.975 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b51b59a9:::benchmark_data_smithi138_477915_object159:head# 2024-09-15T05:47:29.975 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b52536fd:::benchmark_data_smithi138_477915_object102:head# 2024-09-15T05:47:29.976 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b711d581:::benchmark_data_smithi138_477915_object167:head# 2024-09-15T05:47:29.976 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b7914ecd:::benchmark_data_smithi138_477915_object343:head# 2024-09-15T05:47:29.977 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b7ce732e:::benchmark_data_smithi138_477915_object357:head# 2024-09-15T05:47:29.977 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8000b14:::benchmark_data_smithi138_477915_object103:head# 2024-09-15T05:47:29.978 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:47:29.978 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b89297e9:::benchmark_data_smithi138_477915_object362:head# 2024-09-15T05:47:29.979 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b895e95c:::benchmark_data_smithi138_477915_object359:head# 2024-09-15T05:47:29.979 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b897bd1c:::benchmark_data_smithi138_477915_object226:head# 2024-09-15T05:47:29.979 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8f15cb8:::benchmark_data_smithi138_477915_object136:head# 2024-09-15T05:47:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bacecf10:::benchmark_data_smithi138_477915_object340:head# 2024-09-15T05:47:29.980 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bcda79f2:::benchmark_data_smithi138_477915_object228:head# 2024-09-15T05:47:29.981 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bdbfac57:::benchmark_data_smithi138_477915_object171:head# 2024-09-15T05:47:29.981 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:be946f7d:::benchmark_data_smithi138_477915_object390:head# 2024-09-15T05:47:29.982 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bf91c211:::benchmark_data_smithi138_477915_object162:head# 2024-09-15T05:47:29.982 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c080fdda:::benchmark_data_smithi138_477915_object354:head# 2024-09-15T05:47:29.983 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c11f897f:::benchmark_data_smithi138_477915_object236:head# 2024-09-15T05:47:29.983 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c122f290:::benchmark_data_smithi138_477915_object172:head# 2024-09-15T05:47:29.984 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c1ed322f:::benchmark_data_smithi138_477915_object129:head# 2024-09-15T05:47:29.984 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c2bef58a:::benchmark_data_smithi138_477915_object238:head# 2024-09-15T05:47:29.985 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c2e2e8b4:::benchmark_data_smithi138_477915_object234:head# 2024-09-15T05:47:29.985 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c36b3bbe:::benchmark_data_smithi138_477915_object247:head# 2024-09-15T05:47:29.986 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c46189c0:::benchmark_data_smithi138_477915_object300:head# 2024-09-15T05:47:29.986 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c4c0edd9:::benchmark_data_smithi138_477915_object87:head# 2024-09-15T05:47:29.987 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5229070:::benchmark_data_smithi138_477915_object237:head# 2024-09-15T05:47:29.987 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5aafc40:::benchmark_data_smithi138_477915_object21:head# 2024-09-15T05:47:29.988 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5bcceac:::benchmark_data_smithi138_477915_object314:head# 2024-09-15T05:47:29.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5cf901b:::benchmark_data_smithi138_477915_object69:head# 2024-09-15T05:47:29.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c612e5a0:::benchmark_data_smithi138_477915_object131:head# 2024-09-15T05:47:29.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c686c228:::benchmark_data_smithi138_477915_object383:head# 2024-09-15T05:47:29.990 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c6d1efb1:::benchmark_data_smithi138_477915_object269:head# 2024-09-15T05:47:29.990 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7b09ada:::benchmark_data_smithi138_477915_object350:head# 2024-09-15T05:47:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7e9582c:::benchmark_data_smithi138_477915_object318:head# 2024-09-15T05:47:29.991 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8129485:::benchmark_data_smithi138_477915_object400:head# 2024-09-15T05:47:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c81375d4:::benchmark_data_smithi138_477915_object186:head# 2024-09-15T05:47:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c836a7f2:::benchmark_data_smithi138_477915_object392:head# 2024-09-15T05:47:29.992 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c84b5188:::benchmark_data_smithi138_477915_object282:head# 2024-09-15T05:47:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8f99db7:::benchmark_data_smithi138_477915_object295:head# 2024-09-15T05:47:29.993 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cac2fc5a:::benchmark_data_smithi138_477915_object380:head# 2024-09-15T05:47:29.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cb485fde:::benchmark_data_smithi138_477915_object342:head# 2024-09-15T05:47:29.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cbb3fcfe:::benchmark_data_smithi138_477915_object258:head# 2024-09-15T05:47:29.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd46d84a:::benchmark_data_smithi138_477915_object337:head# 2024-09-15T05:47:29.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd852d6c:::benchmark_data_smithi138_477915_object107:head# 2024-09-15T05:47:29.995 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd8a05cc:::benchmark_data_smithi138_477915_object2:head# 2024-09-15T05:47:29.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cec5b2bb:::benchmark_data_smithi138_477915_object25:head# 2024-09-15T05:47:29.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d0e38665:::benchmark_data_smithi138_477915_object266:head# 2024-09-15T05:47:29.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d165ae0f:::benchmark_data_smithi138_477915_object211:head# 2024-09-15T05:47:29.997 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d20fbe61:::benchmark_data_smithi138_477915_object284:head# 2024-09-15T05:47:29.997 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d350cfaa:::benchmark_data_smithi138_477915_object113:head# 2024-09-15T05:47:29.998 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d3b5e940:::benchmark_data_smithi138_477915_object14:head# 2024-09-15T05:47:29.998 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d48aafda:::benchmark_data_smithi138_477915_object358:head# 2024-09-15T05:47:29.999 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d5a212a5:::benchmark_data_smithi138_477915_object106:head# 2024-09-15T05:47:29.999 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d5d6c19c:::benchmark_data_smithi138_477915_object355:head# 2024-09-15T05:47:30.000 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d6d89b5a:::benchmark_data_smithi138_477915_object341:head# 2024-09-15T05:47:30.000 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d789c5f7:::benchmark_data_smithi138_477915_object336:head# 2024-09-15T05:47:30.001 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7c4ab31:::benchmark_data_smithi138_477915_object189:head# 2024-09-15T05:47:30.001 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7c508f3:::benchmark_data_smithi138_477915_object4:head# 2024-09-15T05:47:30.002 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7ebcc2e:::benchmark_data_smithi138_477915_object265:head# 2024-09-15T05:47:30.002 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8daf539:::benchmark_data_smithi138_477915_object99:head# 2024-09-15T05:47:30.003 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8e85bad:::benchmark_data_smithi138_477915_object166:head# 2024-09-15T05:47:30.003 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d95dedd9:::benchmark_data_smithi138_477915_object274:head# 2024-09-15T05:47:30.004 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da2f71a7:::benchmark_data_smithi138_477915_object65:head# 2024-09-15T05:47:30.005 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da407726:::benchmark_data_smithi138_477915_object179:head# 2024-09-15T05:47:30.005 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dabfd9f7:::benchmark_data_smithi138_477915_object118:head# 2024-09-15T05:47:30.005 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dad5a76a:::benchmark_data_smithi138_477915_object305:head# 2024-09-15T05:47:30.006 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd6aea31:::benchmark_data_smithi138_477915_object150:head# 2024-09-15T05:47:30.006 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd78a891:::benchmark_data_smithi138_477915_object353:head# 2024-09-15T05:47:30.007 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dfac43cc:::benchmark_data_smithi138_477915_object126:head# 2024-09-15T05:47:30.007 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dfe8aa68:::benchmark_data_smithi138_477915_object51:head# 2024-09-15T05:47:30.008 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e100c584:::benchmark_data_smithi138_477915_object8:head# 2024-09-15T05:47:30.008 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e1828df5:::benchmark_data_smithi138_477915_object267:head# 2024-09-15T05:47:30.008 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e1daa4ae:::benchmark_data_smithi138_477915_object37:head# 2024-09-15T05:47:30.009 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e23cd4da:::benchmark_data_smithi138_477915_object32:head# 2024-09-15T05:47:30.010 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e4da1a27:::benchmark_data_smithi138_477915_object15:head# 2024-09-15T05:47:30.010 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e4f1a68f:::benchmark_data_smithi138_477915_object395:head# 2024-09-15T05:47:30.010 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e6fdeca3:::benchmark_data_smithi138_477915_object338:head# 2024-09-15T05:47:30.011 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e7c862ab:::benchmark_data_smithi138_477915_object119:head# 2024-09-15T05:47:30.011 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e7e5da90:::benchmark_data_smithi138_477915_object393:head# 2024-09-15T05:47:30.012 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e9e1dfd6:::benchmark_data_smithi138_477915_object315:head# 2024-09-15T05:47:30.012 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eaee7ade:::benchmark_data_smithi138_477915_object277:head# 2024-09-15T05:47:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eb2d0c81:::benchmark_data_smithi138_477915_object352:head# 2024-09-15T05:47:30.013 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eb74e80f:::benchmark_data_smithi138_477915_object39:head# 2024-09-15T05:47:30.014 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ebd302e1:::benchmark_data_smithi138_477915_object221:head# 2024-09-15T05:47:30.014 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec96583a:::benchmark_data_smithi138_477915_object6:head# 2024-09-15T05:47:30.014 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ecd13f7c:::benchmark_data_smithi138_477915_object174:head# 2024-09-15T05:47:30.015 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ecda3af5:::benchmark_data_smithi138_477915_object7:head# 2024-09-15T05:47:30.015 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed2124e8:::benchmark_data_smithi138_477915_object283:head# 2024-09-15T05:47:30.016 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed4f8904:::benchmark_data_smithi138_477915_object321:head# 2024-09-15T05:47:30.016 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eec767c6:::benchmark_data_smithi138_477915_object304:head# 2024-09-15T05:47:30.017 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef20ff5b:::benchmark_data_smithi138_477915_object148:head# 2024-09-15T05:47:30.017 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef304116:::benchmark_data_smithi138_477915_object49:head# 2024-09-15T05:47:30.018 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef573daf:::benchmark_data_smithi138_477915_object177:head# 2024-09-15T05:47:30.018 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f0690f60:::benchmark_data_smithi138_477915_object397:head# 2024-09-15T05:47:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f15d302a:::benchmark_data_smithi138_477915_object81:head# 2024-09-15T05:47:30.019 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f2b2bc3f:::benchmark_data_smithi138_477915_object216:head# 2024-09-15T05:47:30.020 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f314e879:::benchmark_data_smithi138_477915_object290:head# 2024-09-15T05:47:30.020 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f358deca:::benchmark_data_smithi138_477915_object278:head# 2024-09-15T05:47:30.021 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f40ce7f6:::benchmark_data_smithi138_477915_object9:head# 2024-09-15T05:47:30.021 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f48dff42:::benchmark_data_smithi138_477915_object372:head# 2024-09-15T05:47:30.022 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4a59f6a:::benchmark_data_smithi138_477915_object12:head# 2024-09-15T05:47:30.022 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4d52c7d:::benchmark_data_smithi138_477915_object183:head# 2024-09-15T05:47:30.023 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f50e2ea1:::benchmark_data_smithi138_477915_object194:head# 2024-09-15T05:47:30.023 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5165677:::benchmark_data_smithi138_477915_object346:head# 2024-09-15T05:47:30.023 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5ba49b3:::benchmark_data_smithi138_477915_object333:head# 2024-09-15T05:47:30.024 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5c42dd1:::benchmark_data_smithi138_477915_object85:head# 2024-09-15T05:47:30.024 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f64f6100:::benchmark_data_smithi138_477915_object163:head# 2024-09-15T05:47:30.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f6daa659:::benchmark_data_smithi138_477915_object250:head# 2024-09-15T05:47:30.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f7f49d03:::benchmark_data_smithi138_477915_object324:head# 2024-09-15T05:47:30.026 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa26fae3:::benchmark_data_smithi138_477915_object73:head# 2024-09-15T05:47:30.027 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa6cc2ae:::benchmark_data_smithi138_477915_object91:head# 2024-09-15T05:47:30.028 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa9ab79f:::benchmark_data_smithi138_477915_object72:head# 2024-09-15T05:47:30.028 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fb6a8c6a:::benchmark_data_smithi138_477915_object122:head# 2024-09-15T05:47:30.029 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fba968a6:::benchmark_data_smithi138_477915_object368:head# 2024-09-15T05:47:30.029 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fd402ec6:::benchmark_data_smithi138_477915_object67:head# 2024-09-15T05:47:30.030 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fdef6d22:::benchmark_data_smithi138_477915_object309:head# 2024-09-15T05:47:30.030 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fe273b33:::benchmark_data_smithi138_477915_object146:head# 2024-09-15T05:47:30.031 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fea7c25b:::benchmark_data_smithi138_477915_object344:head# 2024-09-15T05:47:30.031 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ff1959bb:::benchmark_data_smithi138_477915_object80:head# 2024-09-15T05:47:30.032 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffb3fc0b:::benchmark_data_smithi138_477915_object115:head# 2024-09-15T05:47:30.032 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffbe43c0:::benchmark_data_smithi138_477915_object125:head# 2024-09-15T05:47:30.033 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T05:47:30.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-15T05:47:31.154 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.0 info 1.0( v 9'209 (0'0,9'209] local-lis/les=7/8 n=209 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:47:31.174 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:004eb6a1:::benchmark_data_smithi138_477915_object24:head# 2024-09-15T05:47:31.191 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:01723c07:::benchmark_data_smithi138_477915_object335:head# 2024-09-15T05:47:31.207 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:021e9c3c:::benchmark_data_smithi138_477915_object60:head# 2024-09-15T05:47:31.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0276aebd:::benchmark_data_smithi138_477915_object108:head# 2024-09-15T05:47:31.224 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:03123a40:::benchmark_data_smithi138_477915_object176:head# 2024-09-15T05:47:31.241 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:031b6711:::benchmark_data_smithi138_477915_object198:head# 2024-09-15T05:47:31.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0432e19c:::benchmark_data_smithi138_477915_object62:head# 2024-09-15T05:47:31.265 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:04a09b67:::benchmark_data_smithi138_477915_object401:head# 2024-09-15T05:47:31.266 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:04af6a20:::benchmark_data_smithi138_477915_object98:head# 2024-09-15T05:47:31.291 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05c5c04d:::benchmark_data_smithi138_477915_object195:head# 2024-09-15T05:47:31.299 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05ce29b1:::benchmark_data_smithi138_477915_object181:head# 2024-09-15T05:47:31.317 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05d45069:::benchmark_data_smithi138_477915_object275:head# 2024-09-15T05:47:31.333 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:07351a71:::benchmark_data_smithi138_477915_object100:head# 2024-09-15T05:47:31.341 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:07fbc7df:::benchmark_data_smithi138_477915_object312:head# 2024-09-15T05:47:31.349 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08079dbc:::benchmark_data_smithi138_477915_object299:head# 2024-09-15T05:47:31.350 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08859349:::benchmark_data_smithi138_477915_object223:head# 2024-09-15T05:47:31.366 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08a0ce86:::benchmark_data_smithi138_477915_object385:head# 2024-09-15T05:47:31.375 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08baf2ed:::benchmark_data_smithi138_477915_object192:head# 2024-09-15T05:47:31.376 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08bd9a02:::benchmark_data_smithi138_477915_object214:head# 2024-09-15T05:47:31.391 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:097fa44d:::benchmark_data_smithi138_477915_object328:head# 2024-09-15T05:47:31.392 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a0d4136:::benchmark_data_smithi138_477915_object30:head# 2024-09-15T05:47:31.400 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a2641dd:::benchmark_data_smithi138_477915_object210:head# 2024-09-15T05:47:31.401 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a2d6d17:::benchmark_data_smithi138_477915_object61:head# 2024-09-15T05:47:31.417 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ac84968:::benchmark_data_smithi138_477915_object286:head# 2024-09-15T05:47:31.425 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ad209ab:::benchmark_data_smithi138_477915_object70:head# 2024-09-15T05:47:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0b64715f:::benchmark_data_smithi138_477915_object17:head# 2024-09-15T05:47:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0bbf6561:::benchmark_data_smithi138_477915_object137:head# 2024-09-15T05:47:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0cca7b5b:::benchmark_data_smithi138_477915_object339:head# 2024-09-15T05:47:31.426 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0d194a0a:::benchmark_data_smithi138_477915_object77:head# 2024-09-15T05:47:31.427 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0e135564:::benchmark_data_smithi138_477915_object313:head# 2024-09-15T05:47:31.427 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0e15c428:::benchmark_data_smithi138_477915_object120:head# 2024-09-15T05:47:31.428 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0f4410e4:::benchmark_data_smithi138_477915_object145:head# 2024-09-15T05:47:31.428 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0fefd637:::benchmark_data_smithi138_477915_object389:head# 2024-09-15T05:47:31.429 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10535269:::benchmark_data_smithi138_477915_object332:head# 2024-09-15T05:47:31.429 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10bc54c8:::benchmark_data_smithi138_477915_object225:head# 2024-09-15T05:47:31.430 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10e6b441:::benchmark_data_smithi138_477915_object248:head# 2024-09-15T05:47:31.430 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1161a4f5:::benchmark_data_smithi138_477915_object281:head# 2024-09-15T05:47:31.431 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:11ab1fa3:::benchmark_data_smithi138_477915_object240:head# 2024-09-15T05:47:31.431 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:11b92b78:::benchmark_data_smithi138_477915_object142:head# 2024-09-15T05:47:31.431 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:13efdddb:::benchmark_data_smithi138_477915_object57:head# 2024-09-15T05:47:31.432 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:14c54418:::benchmark_data_smithi138_477915_object197:head# 2024-09-15T05:47:31.432 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:152a39dd:::benchmark_data_smithi138_477915_object144:head# 2024-09-15T05:47:31.433 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15322a7a:::benchmark_data_smithi138_477915_object288:head# 2024-09-15T05:47:31.433 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15aaf075:::benchmark_data_smithi138_477915_object128:head# 2024-09-15T05:47:31.434 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16dad5c5:::benchmark_data_smithi138_477915_object252:head# 2024-09-15T05:47:31.438 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16f17ac6:::benchmark_data_smithi138_477915_object329:head# 2024-09-15T05:47:31.438 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1708f8e0:::benchmark_data_smithi138_477915_object307:head# 2024-09-15T05:47:31.438 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1854249c:::benchmark_data_smithi138_477915_object46:head# 2024-09-15T05:47:31.438 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:191dd67b:::benchmark_data_smithi138_477915_object20:head# 2024-09-15T05:47:31.438 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1acb70bd:::benchmark_data_smithi138_477915_object19:head# 2024-09-15T05:47:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b1b6402:::benchmark_data_smithi138_477915_object196:head# 2024-09-15T05:47:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b7cd855:::benchmark_data_smithi138_477915_object243:head# 2024-09-15T05:47:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1baea71e:::benchmark_data_smithi138_477915_object330:head# 2024-09-15T05:47:31.439 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1c0c4030:::benchmark_data_smithi138_477915_object325:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1ced30b9:::benchmark_data_smithi138_477915_object165:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1dac8c2e:::benchmark_data_smithi138_477915_object204:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1dc29afc:::benchmark_data_smithi138_477915_object398:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1f7be81b:::benchmark_data_smithi138_477915_object292:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20000fdb:::benchmark_data_smithi138_477915_object157:head# 2024-09-15T05:47:31.441 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20ce187a:::benchmark_data_smithi138_477915_object116:head# 2024-09-15T05:47:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:211ff0cd:::benchmark_data_smithi138_477915_object209:head# 2024-09-15T05:47:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:214fff42:::benchmark_data_smithi138_477915_object230:head# 2024-09-15T05:47:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:219197a2:::benchmark_data_smithi138_477915_object111:head# 2024-09-15T05:47:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:22101c95:::benchmark_data_smithi138_477915_object59:head# 2024-09-15T05:47:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:223f5228:::benchmark_data_smithi138_477915_object79:head# 2024-09-15T05:47:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:230de784:::benchmark_data_smithi138_477915_object244:head# 2024-09-15T05:47:31.444 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:246b4656:::benchmark_data_smithi138_477915_object191:head# 2024-09-15T05:47:31.445 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:25092a54:::benchmark_data_smithi138_477915_object10:head# 2024-09-15T05:47:31.445 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:263d2c7b:::benchmark_data_smithi138_477915_object153:head# 2024-09-15T05:47:31.445 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:266b4db5:::benchmark_data_smithi138_477915_object13:head# 2024-09-15T05:47:31.446 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26e68133:::benchmark_data_smithi138_477915_object349:head# 2024-09-15T05:47:31.446 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26f80018:::benchmark_data_smithi138_477915_object75:head# 2024-09-15T05:47:31.447 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2729df23:::benchmark_data_smithi138_477915_object245:head# 2024-09-15T05:47:31.447 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:273a39c8:::benchmark_data_smithi138_477915_object212:head# 2024-09-15T05:47:31.447 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b21cb84:::benchmark_data_smithi138_477915_object366:head# 2024-09-15T05:47:31.448 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2e0a6ba0:::benchmark_data_smithi138_477915_object45:head# 2024-09-15T05:47:31.449 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2fd3771a:::benchmark_data_smithi138_477915_object326:head# 2024-09-15T05:47:31.449 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:30de888f:::benchmark_data_smithi138_477915_object34:head# 2024-09-15T05:47:31.450 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:312dc4e7:::benchmark_data_smithi138_477915_object232:head# 2024-09-15T05:47:31.450 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:318ba285:::benchmark_data_smithi138_477915_object164:head# 2024-09-15T05:47:31.450 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:31e1d00e:::benchmark_data_smithi138_477915_object44:head# 2024-09-15T05:47:31.452 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:32a36681:::benchmark_data_smithi138_477915_object90:head# 2024-09-15T05:47:31.452 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3397aa63:::benchmark_data_smithi138_477915_object294:head# 2024-09-15T05:47:31.452 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:350297ef:::benchmark_data_smithi138_477915_object382:head# 2024-09-15T05:47:31.453 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:354f77ae:::benchmark_data_smithi138_477915_object378:head# 2024-09-15T05:47:31.453 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:359485cb:::benchmark_data_smithi138_477915_object363:head# 2024-09-15T05:47:31.453 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:386dd901:::benchmark_data_smithi138_477915_object287:head# 2024-09-15T05:47:31.454 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:389afc62:::benchmark_data_smithi138_477915_object206:head# 2024-09-15T05:47:31.455 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38edce5f:::benchmark_data_smithi138_477915_object202:head# 2024-09-15T05:47:31.455 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:392023cd:::benchmark_data_smithi138_477915_object360:head# 2024-09-15T05:47:31.456 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:396a3af9:::benchmark_data_smithi138_477915_object361:head# 2024-09-15T05:47:31.456 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:39870a18:::benchmark_data_smithi138_477915_object55:head# 2024-09-15T05:47:31.457 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3a34f987:::benchmark_data_smithi138_477915_object84:head# 2024-09-15T05:47:31.457 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3a45d3f0:::benchmark_data_smithi138_477915_object133:head# 2024-09-15T05:47:31.457 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3bc519ab:::benchmark_data_smithi138_477915_object301:head# 2024-09-15T05:47:31.458 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3bc57672:::benchmark_data_smithi138_477915_object317:head# 2024-09-15T05:47:31.458 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3d290d02:::benchmark_data_smithi138_477915_object52:head# 2024-09-15T05:47:31.458 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3d995ed5:::benchmark_data_smithi138_477915_object289:head# 2024-09-15T05:47:31.459 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3e073204:::benchmark_data_smithi138_477915_object47:head# 2024-09-15T05:47:31.459 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3f24c379:::benchmark_data_smithi138_477915_object381:head# 2024-09-15T05:47:31.460 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3f8ef9b0:::benchmark_data_smithi138_477915_object215:head# 2024-09-15T05:47:31.461 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4188c502:::benchmark_data_smithi138_477915_object394:head# 2024-09-15T05:47:31.461 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:41d0e70f:::benchmark_data_smithi138_477915_object124:head# 2024-09-15T05:47:31.462 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:42ce4ca2:::benchmark_data_smithi138_477915_object170:head# 2024-09-15T05:47:31.464 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4384dd5f:::benchmark_data_smithi138_477915_object222:head# 2024-09-15T05:47:31.464 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:441e81fc:::benchmark_data_smithi138_477915_object322:head# 2024-09-15T05:47:31.464 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:444f377d:::benchmark_data_smithi138_477915_object259:head# 2024-09-15T05:47:31.465 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44d1d1a6:::benchmark_data_smithi138_477915_object175:head# 2024-09-15T05:47:31.465 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44e95b23:::benchmark_data_smithi138_477915_object187:head# 2024-09-15T05:47:31.465 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44f216b2:::benchmark_data_smithi138_477915_object331:head# 2024-09-15T05:47:31.465 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:45f59372:::benchmark_data_smithi138_477915_object134:head# 2024-09-15T05:47:31.468 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:463b99f0:::benchmark_data_smithi138_477915_object41:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:46f42845:::benchmark_data_smithi138_477915_object50:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:47847711:::benchmark_data_smithi138_477915_object391:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:481a4dcf:::benchmark_data_smithi138_477915_object306:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48339ebf:::benchmark_data_smithi138_477915_object53:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48cd125a:::benchmark_data_smithi138_477915_object320:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:49297529:::benchmark_data_smithi138_477915_object110:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:492bb000:::benchmark_data_smithi138_477915_object364:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4a0a3214:::benchmark_data_smithi138_477915_object220:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4b9543c5:::benchmark_data_smithi138_477915_object1:head# 2024-09-15T05:47:31.469 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4ba0927c:::benchmark_data_smithi138_477915_object178:head# 2024-09-15T05:47:31.470 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4ba2f324:::benchmark_data_smithi138_477915_object293:head# 2024-09-15T05:47:31.470 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4c0b6957:::benchmark_data_smithi138_477915_object114:head# 2024-09-15T05:47:31.470 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4c7779cc:::benchmark_data_smithi138_477915_object379:head# 2024-09-15T05:47:31.471 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4cffde3a:::benchmark_data_smithi138_477915_object271:head# 2024-09-15T05:47:31.471 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4d3925ed:::benchmark_data_smithi138_477915_object396:head# 2024-09-15T05:47:31.472 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4d67e2bb:::benchmark_data_smithi138_477915_object141:head# 2024-09-15T05:47:31.472 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4e93ce33:::benchmark_data_smithi138_477915_object82:head# 2024-09-15T05:47:31.472 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4eb39cc3:::benchmark_data_smithi138_477915_object11:head# 2024-09-15T05:47:31.473 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4fdb5ea1:::benchmark_data_smithi138_477915_object86:head# 2024-09-15T05:47:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52b28e51:::benchmark_data_smithi138_477915_object188:head# 2024-09-15T05:47:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52f1d98b:::benchmark_data_smithi138_477915_object29:head# 2024-09-15T05:47:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:538d02e4:::benchmark_data_smithi138_477915_object387:head# 2024-09-15T05:47:31.474 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:53b28f04:::benchmark_data_smithi138_477915_object193:head# 2024-09-15T05:47:31.475 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:543e73a1:::benchmark_data_smithi138_477915_object143:head# 2024-09-15T05:47:31.475 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:546413ea:::benchmark_data_smithi138_477915_object35:head# 2024-09-15T05:47:31.476 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54b5d110:::benchmark_data_smithi138_477915_object370:head# 2024-09-15T05:47:31.476 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54d808a1:::benchmark_data_smithi138_477915_object147:head# 2024-09-15T05:47:31.477 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:556f515c:::benchmark_data_smithi138_477915_object367:head# 2024-09-15T05:47:31.477 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5581a3b9:::benchmark_data_smithi138_477915_object123:head# 2024-09-15T05:47:31.478 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:55b4ce8c:::benchmark_data_smithi138_477915_object327:head# 2024-09-15T05:47:31.479 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:55e0f391:::benchmark_data_smithi138_477915_object40:head# 2024-09-15T05:47:31.479 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:564dfa9a:::benchmark_data_smithi138_477915_object89:head# 2024-09-15T05:47:31.480 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56619472:::benchmark_data_smithi138_477915_object291:head# 2024-09-15T05:47:31.480 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56ac2e18:::benchmark_data_smithi138_477915_object373:head# 2024-09-15T05:47:31.480 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56e0ff0d:::benchmark_data_smithi138_477915_object246:head# 2024-09-15T05:47:31.481 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56e54ad7:::benchmark_data_smithi138_477915_object241:head# 2024-09-15T05:47:31.481 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:572a26cc:::benchmark_data_smithi138_477915_object3:head# 2024-09-15T05:47:31.481 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:57365c69:::benchmark_data_smithi138_477915_object185:head# 2024-09-15T05:47:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:58b05a98:::benchmark_data_smithi138_477915_object94:head# 2024-09-15T05:47:31.482 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5979f860:::benchmark_data_smithi138_477915_object42:head# 2024-09-15T05:47:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5a3930c3:::benchmark_data_smithi138_477915_object92:head# 2024-09-15T05:47:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5acdf762:::benchmark_data_smithi138_477915_object217:head# 2024-09-15T05:47:31.483 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5b192008:::benchmark_data_smithi138_477915_object190:head# 2024-09-15T05:47:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5bd647c9:::benchmark_data_smithi138_477915_object308:head# 2024-09-15T05:47:31.484 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5c5abf37:::benchmark_data_smithi138_477915_object227:head# 2024-09-15T05:47:31.485 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5cc75e94:::benchmark_data_smithi138_477915_object261:head# 2024-09-15T05:47:31.485 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5d452870:::benchmark_data_smithi138_477915_object386:head# 2024-09-15T05:47:31.486 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5e2e9211:::benchmark_data_smithi138_477915_object201:head# 2024-09-15T05:47:31.486 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5f10ef7e:::benchmark_data_smithi138_477915_object182:head# 2024-09-15T05:47:31.486 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fa8bc7a:::benchmark_data_smithi138_477915_object371:head# 2024-09-15T05:47:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fd3194b:::benchmark_data_smithi138_477915_object83:head# 2024-09-15T05:47:31.487 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fec7540:::benchmark_data_smithi138_477915_object132:head# 2024-09-15T05:47:31.488 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6012bbb1:::benchmark_data_smithi138_477915_object213:head# 2024-09-15T05:47:31.490 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:620660f2:::benchmark_data_smithi138_477915_object138:head# 2024-09-15T05:47:31.490 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:62fcea3a:::benchmark_data_smithi138_477915_object374:head# 2024-09-15T05:47:31.490 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:635fcc02:::benchmark_data_smithi138_477915_object18:head# 2024-09-15T05:47:31.490 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6429912b:::benchmark_data_smithi138_477915_object78:head# 2024-09-15T05:47:31.490 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:648cc3c2:::benchmark_data_smithi138_477915_object180:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:64ae8f93:::benchmark_data_smithi138_477915_object273:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:65851d9e:::benchmark_data_smithi138_477915_object276:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:66acc5ee:::benchmark_data_smithi138_477915_object155:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:67e15da6:::benchmark_data_smithi138_477915_object253:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68a5ecca:::benchmark_data_smithi138_477915_object260:head# 2024-09-15T05:47:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68b33428:::benchmark_data_smithi138_477915_object388:head# 2024-09-15T05:47:31.494 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6940b9d3:::benchmark_data_smithi138_477915_object239:head# 2024-09-15T05:47:31.494 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:69f7ef4c:::benchmark_data_smithi138_477915_object127:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6aef21a2:::benchmark_data_smithi138_477915_object54:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6b3c2b63:::benchmark_data_smithi138_477915_object208:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6bf4848f:::benchmark_data_smithi138_477915_object399:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6ccf34bf:::benchmark_data_smithi138_477915_object22:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6d5a3232:::benchmark_data_smithi138_477915_object43:head# 2024-09-15T05:47:31.495 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dc7f41b:::benchmark_data_smithi138_477915_object262:head# 2024-09-15T05:47:31.496 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dd17fb9:::benchmark_data_smithi138_477915_object263:head# 2024-09-15T05:47:31.497 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6ddcb643:::benchmark_data_smithi138_477915_object285:head# 2024-09-15T05:47:31.497 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6e1716aa:::benchmark_data_smithi138_477915_object135:head# 2024-09-15T05:47:31.497 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:702ed571:::benchmark_data_smithi138_477915_object219:head# 2024-09-15T05:47:31.498 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:707bb828:::benchmark_data_smithi138_477915_object158:head# 2024-09-15T05:47:31.498 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:70ca4348:::benchmark_data_smithi138_477915_object26:head# 2024-09-15T05:47:31.498 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:71784a9a:::benchmark_data_smithi138_477915_object376:head# 2024-09-15T05:47:31.498 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:717dd36a:::benchmark_data_smithi138_477915_object139:head# 2024-09-15T05:47:31.499 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:71fb5440:::benchmark_data_smithi138_477915_object97:head# 2024-09-15T05:47:31.499 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7314c8d4:::benchmark_data_smithi138_477915_object152:head# 2024-09-15T05:47:31.500 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:740159d0:::benchmark_data_smithi138_477915_object0:head# 2024-09-15T05:47:31.500 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:74886e53:::benchmark_data_smithi138_477915_object112:head# 2024-09-15T05:47:31.500 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:75046150:::benchmark_data_smithi138_477915_object279:head# 2024-09-15T05:47:31.501 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:773e28fe:::benchmark_data_smithi138_477915_object377:head# 2024-09-15T05:47:31.501 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7783b60c:::benchmark_data_smithi138_477915_object101:head# 2024-09-15T05:47:31.502 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:786160e4:::benchmark_data_smithi138_477915_object38:head# 2024-09-15T05:47:31.502 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7a2bef6b:::benchmark_data_smithi138_477915_object58:head# 2024-09-15T05:47:31.503 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7a76feec:::benchmark_data_smithi138_477915_object63:head# 2024-09-15T05:47:31.503 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7bb84b6a:::benchmark_data_smithi138_477915_object5:head# 2024-09-15T05:47:31.503 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7be49943:::benchmark_data_smithi138_477915_object48:head# 2024-09-15T05:47:31.504 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d976229:::benchmark_data_smithi138_477915_object207:head# 2024-09-15T05:47:31.504 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d9d7a5f:::benchmark_data_smithi138_477915_object369:head# 2024-09-15T05:47:31.504 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7eaec87b:::benchmark_data_smithi138_477915_object154:head# 2024-09-15T05:47:31.505 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7eda30d0:::benchmark_data_smithi138_477915_object310:head# 2024-09-15T05:47:31.505 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7f744066:::benchmark_data_smithi138_477915_object205:head# 2024-09-15T05:47:31.506 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:47:31.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:47:31.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:47:31.777 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:47:31.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:47:31.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:47:31.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:47:31.780 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:47:31.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:47:31.801 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:47:31.800+0000 7f64be273440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:47:31.803 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:47:31.803+0000 7f64be273440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:47:31.805 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:47:31.804+0000 7f64be273440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:47:32.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:47:32.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:47:32.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:47:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:47:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:47:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:32.131 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:47:32.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:47:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:32.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:32.322 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:47:32.321+0000 7f64be273440 -1 Falling back to public interface 2024-09-15T05:47:32.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:33.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:33.516 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:47:33.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:33.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:47:33.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:33.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:33.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:47:34.668 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:47:34.667+0000 7f64be273440 -1 osd.0 9 log_to_monitors true 2024-09-15T05:47:34.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:47:34.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:47:34.919 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:47:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:47:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:47:34.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:47:35.320 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/478730,v1:127.0.0.1:6803/478730] [v2:127.0.0.1:6804/478730,v1:127.0.0.1:6805/478730] exists,up be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:47:35.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:47:35.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:47:35.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:47:35.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-09-15T05:47:36.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-09-15T05:47:41.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-09-15T05:47:41.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-09-15T05:47:41.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-09-15T05:47:41.096 INFO:tasks.workunit.client.0.smithi138.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T05:47:41.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-15T05:47:41.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:47:41.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:47:41.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:47:41.098 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:47:41.099 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:47:41.099 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:47:41.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:47:41.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:47:41.099 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:47:41.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:47:41.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:47:41.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:47:41.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:47:41.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:47:41.187 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:47:41.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:47:41.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:47:41.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:47:41.535 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:47:41.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607556 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607556 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-51539607556' 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-51539607556 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:47:41.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:47:41.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-51539607556 2024-09-15T05:47:41.653 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:47:41.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607556 2024-09-15T05:47:41.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-15T05:47:41.654 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 51539607556 2024-09-15T05:47:41.654 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:47:42.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-15T05:47:42.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:47:43.085 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:47:43.085 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:47:43.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-15T05:47:43.454 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:47:43.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:47:43.455 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:47:43.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:47:43.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:47:43.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:47:44.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:47:44.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:47:44.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:47:44.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:47:44.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:47:45.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:47:45.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-15T05:47:46.283 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:47:46.283 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:47:46.283 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T05:47:46.793 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:47:47.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-15T05:47:48.231 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-15T05:47:48.231 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.1 2024-09-15T05:47:48.232 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 2 -> 1 2024-09-15T05:47:48.232 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81cead58:::benchmark_data_smithi138_477915_object95:head# 2024-09-15T05:47:48.232 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.372 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81e31d0f:::benchmark_data_smithi138_477915_object203:head# 2024-09-15T05:47:48.372 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:829c9f42:::benchmark_data_smithi138_477915_object297:head# 2024-09-15T05:47:48.422 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.472 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8326356e:::benchmark_data_smithi138_477915_object272:head# 2024-09-15T05:47:48.472 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.531 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84ba03a2:::benchmark_data_smithi138_477915_object268:head# 2024-09-15T05:47:48.531 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.598 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84bb0071:::benchmark_data_smithi138_477915_object169:head# 2024-09-15T05:47:48.598 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.668 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8531dfd7:::benchmark_data_smithi138_477915_object311:head# 2024-09-15T05:47:48.668 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.766 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:853bf743:::benchmark_data_smithi138_477915_object257:head# 2024-09-15T05:47:48.766 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.824 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e0b007:::benchmark_data_smithi138_477915_object74:head# 2024-09-15T05:47:48.824 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.858 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e93436:::benchmark_data_smithi138_477915_object319:head# 2024-09-15T05:47:48.858 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.891 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8709604b:::benchmark_data_smithi138_477915_object347:head# 2024-09-15T05:47:48.891 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.941 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:87d6852e:::benchmark_data_smithi138_477915_object231:head# 2024-09-15T05:47:48.942 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:48.975 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:88c42840:::benchmark_data_smithi138_477915_object140:head# 2024-09-15T05:47:48.975 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.008 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:89a81864:::benchmark_data_smithi138_477915_object264:head# 2024-09-15T05:47:49.009 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.033 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a16f6c1:::benchmark_data_smithi138_477915_object93:head# 2024-09-15T05:47:49.034 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.059 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aa67a0a:::benchmark_data_smithi138_477915_object235:head# 2024-09-15T05:47:49.059 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.084 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aedb92d:::benchmark_data_smithi138_477915_object365:head# 2024-09-15T05:47:49.084 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.109 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8be95c11:::benchmark_data_smithi138_477915_object323:head# 2024-09-15T05:47:49.109 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.134 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c3675a7:::benchmark_data_smithi138_477915_object303:head# 2024-09-15T05:47:49.134 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.159 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d8d0588:::benchmark_data_smithi138_477915_object28:head# 2024-09-15T05:47:49.160 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.226 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f046a5e:::benchmark_data_smithi138_477915_object109:head# 2024-09-15T05:47:49.226 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.277 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f04eb67:::benchmark_data_smithi138_477915_object160:head# 2024-09-15T05:47:49.277 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.319 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:900c1e9a:::benchmark_data_smithi138_477915_object168:head# 2024-09-15T05:47:49.319 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.352 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9033da55:::benchmark_data_smithi138_477915_object23:head# 2024-09-15T05:47:49.352 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.385 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90dfb114:::benchmark_data_smithi138_477915_object233:head# 2024-09-15T05:47:49.386 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.419 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90f41851:::benchmark_data_smithi138_477915_object256:head# 2024-09-15T05:47:49.419 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.453 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91c6592f:::benchmark_data_smithi138_477915_object36:head# 2024-09-15T05:47:49.453 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.486 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91f02d94:::benchmark_data_smithi138_477915_object249:head# 2024-09-15T05:47:49.486 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.520 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92efb357:::benchmark_data_smithi138_477915_object242:head# 2024-09-15T05:47:49.520 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.553 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9432f393:::benchmark_data_smithi138_477915_object16:head# 2024-09-15T05:47:49.553 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.595 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971a84a9:::benchmark_data_smithi138_477915_object149:head# 2024-09-15T05:47:49.595 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.629 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971af863:::benchmark_data_smithi138_477915_object151:head# 2024-09-15T05:47:49.629 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.662 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a712198:::benchmark_data_smithi138_477915_object218:head# 2024-09-15T05:47:49.662 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.695 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9df1f34f:::benchmark_data_smithi138_477915_object356:head# 2024-09-15T05:47:49.696 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.729 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f9adf98:::benchmark_data_smithi138_477915_object200:head# 2024-09-15T05:47:49.729 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.762 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a080aed2:::benchmark_data_smithi138_477915_object104:head# 2024-09-15T05:47:49.762 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.796 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0aeaa72:::benchmark_data_smithi138_477915_object254:head# 2024-09-15T05:47:49.796 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:49.962 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0d22f50:::benchmark_data_smithi138_477915_object71:head# 2024-09-15T05:47:49.962 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.079 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10a53f5:::benchmark_data_smithi138_477915_object298:head# 2024-09-15T05:47:50.080 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.113 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10dc166:::benchmark_data_smithi138_477915_object117:head# 2024-09-15T05:47:50.113 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.147 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1460743:::benchmark_data_smithi138_477915_object334:head# 2024-09-15T05:47:50.147 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.180 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a14ca631:::benchmark_data_smithi138_477915_object105:head# 2024-09-15T05:47:50.180 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.213 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1c26988:::benchmark_data_smithi138_477915_object351:head# 2024-09-15T05:47:50.214 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.247 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a2e8a8e9:::benchmark_data_smithi138_477915_object184:head# 2024-09-15T05:47:50.247 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.280 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5019f41:::benchmark_data_smithi138_477915_object302:head# 2024-09-15T05:47:50.280 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.314 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a574d19f:::benchmark_data_smithi138_477915_object27:head# 2024-09-15T05:47:50.314 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.347 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5add657:::benchmark_data_smithi138_477915_object96:head# 2024-09-15T05:47:50.348 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.381 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a65ba2e1:::benchmark_data_smithi138_477915_object348:head# 2024-09-15T05:47:50.381 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.415 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a76a8553:::benchmark_data_smithi138_477915_object384:head# 2024-09-15T05:47:50.415 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.448 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a84031a3:::benchmark_data_smithi138_477915_object156:head# 2024-09-15T05:47:50.448 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.481 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a89e1157:::benchmark_data_smithi138_477915_object66:head# 2024-09-15T05:47:50.482 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.524 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a8ec4680:::benchmark_data_smithi138_477915_object280:head# 2024-09-15T05:47:50.524 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.557 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a91194b3:::benchmark_data_smithi138_477915_object68:head# 2024-09-15T05:47:50.557 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.590 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aae671fe:::benchmark_data_smithi138_477915_object173:head# 2024-09-15T05:47:50.590 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.624 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:abf706c3:::benchmark_data_smithi138_477915_object121:head# 2024-09-15T05:47:50.624 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.684 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ad8a20c0:::benchmark_data_smithi138_477915_object375:head# 2024-09-15T05:47:50.684 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.733 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ade77a16:::benchmark_data_smithi138_477915_object88:head# 2024-09-15T05:47:50.733 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.783 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ae6263f9:::benchmark_data_smithi138_477915_object31:head# 2024-09-15T05:47:50.783 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.816 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b05f7e87:::benchmark_data_smithi138_477915_object316:head# 2024-09-15T05:47:50.817 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.850 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b0627822:::benchmark_data_smithi138_477915_object255:head# 2024-09-15T05:47:50.850 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.884 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b08f0d28:::benchmark_data_smithi138_477915_object229:head# 2024-09-15T05:47:50.884 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.917 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b1344ffa:::benchmark_data_smithi138_477915_object33:head# 2024-09-15T05:47:50.917 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.950 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b142aea1:::benchmark_data_smithi138_477915_object64:head# 2024-09-15T05:47:50.950 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:50.984 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b53955:::benchmark_data_smithi138_477915_object199:head# 2024-09-15T05:47:50.984 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.017 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b94858:::benchmark_data_smithi138_477915_object345:head# 2024-09-15T05:47:51.017 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.059 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b30ed2d9:::benchmark_data_smithi138_477915_object224:head# 2024-09-15T05:47:51.059 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.093 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b36e2fb6:::benchmark_data_smithi138_477915_object56:head# 2024-09-15T05:47:51.093 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.118 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b388f84c:::benchmark_data_smithi138_477915_object270:head# 2024-09-15T05:47:51.118 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.143 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3bb6a6a:::benchmark_data_smithi138_477915_object251:head# 2024-09-15T05:47:51.143 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.177 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3ceb653:::benchmark_data_smithi138_477915_object130:head# 2024-09-15T05:47:51.177 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.227 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b413d550:::benchmark_data_smithi138_477915_object76:head# 2024-09-15T05:47:51.227 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.286 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b49d13a3:::benchmark_data_smithi138_477915_object296:head# 2024-09-15T05:47:51.286 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.327 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b4dcf7e3:::benchmark_data_smithi138_477915_object161:head# 2024-09-15T05:47:51.328 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.353 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b51b59a9:::benchmark_data_smithi138_477915_object159:head# 2024-09-15T05:47:51.353 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.386 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b52536fd:::benchmark_data_smithi138_477915_object102:head# 2024-09-15T05:47:51.386 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b711d581:::benchmark_data_smithi138_477915_object167:head# 2024-09-15T05:47:51.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.445 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7914ecd:::benchmark_data_smithi138_477915_object343:head# 2024-09-15T05:47:51.445 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.478 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7ce732e:::benchmark_data_smithi138_477915_object357:head# 2024-09-15T05:47:51.478 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.512 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8000b14:::benchmark_data_smithi138_477915_object103:head# 2024-09-15T05:47:51.512 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.545 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:47:51.545 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.578 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b89297e9:::benchmark_data_smithi138_477915_object362:head# 2024-09-15T05:47:51.579 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.612 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b895e95c:::benchmark_data_smithi138_477915_object359:head# 2024-09-15T05:47:51.612 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.646 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b897bd1c:::benchmark_data_smithi138_477915_object226:head# 2024-09-15T05:47:51.646 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.679 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8f15cb8:::benchmark_data_smithi138_477915_object136:head# 2024-09-15T05:47:51.679 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.713 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bacecf10:::benchmark_data_smithi138_477915_object340:head# 2024-09-15T05:47:51.713 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.746 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bcda79f2:::benchmark_data_smithi138_477915_object228:head# 2024-09-15T05:47:51.746 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.779 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bdbfac57:::benchmark_data_smithi138_477915_object171:head# 2024-09-15T05:47:51.780 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.813 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:be946f7d:::benchmark_data_smithi138_477915_object390:head# 2024-09-15T05:47:51.813 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.847 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bf91c211:::benchmark_data_smithi138_477915_object162:head# 2024-09-15T05:47:51.847 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.880 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c080fdda:::benchmark_data_smithi138_477915_object354:head# 2024-09-15T05:47:51.880 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.913 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c11f897f:::benchmark_data_smithi138_477915_object236:head# 2024-09-15T05:47:51.914 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.947 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c122f290:::benchmark_data_smithi138_477915_object172:head# 2024-09-15T05:47:51.947 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:51.980 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c1ed322f:::benchmark_data_smithi138_477915_object129:head# 2024-09-15T05:47:51.981 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.014 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2bef58a:::benchmark_data_smithi138_477915_object238:head# 2024-09-15T05:47:52.014 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.047 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2e2e8b4:::benchmark_data_smithi138_477915_object234:head# 2024-09-15T05:47:52.047 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.081 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c36b3bbe:::benchmark_data_smithi138_477915_object247:head# 2024-09-15T05:47:52.081 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.116 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c46189c0:::benchmark_data_smithi138_477915_object300:head# 2024-09-15T05:47:52.117 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.165 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4c0edd9:::benchmark_data_smithi138_477915_object87:head# 2024-09-15T05:47:52.165 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.198 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5229070:::benchmark_data_smithi138_477915_object237:head# 2024-09-15T05:47:52.199 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.232 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5aafc40:::benchmark_data_smithi138_477915_object21:head# 2024-09-15T05:47:52.232 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.265 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5bcceac:::benchmark_data_smithi138_477915_object314:head# 2024-09-15T05:47:52.265 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.299 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5cf901b:::benchmark_data_smithi138_477915_object69:head# 2024-09-15T05:47:52.299 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.332 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c612e5a0:::benchmark_data_smithi138_477915_object131:head# 2024-09-15T05:47:52.332 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.366 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c686c228:::benchmark_data_smithi138_477915_object383:head# 2024-09-15T05:47:52.366 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.399 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c6d1efb1:::benchmark_data_smithi138_477915_object269:head# 2024-09-15T05:47:52.399 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.433 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7b09ada:::benchmark_data_smithi138_477915_object350:head# 2024-09-15T05:47:52.433 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.466 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7e9582c:::benchmark_data_smithi138_477915_object318:head# 2024-09-15T05:47:52.466 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.500 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8129485:::benchmark_data_smithi138_477915_object400:head# 2024-09-15T05:47:52.500 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.533 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c81375d4:::benchmark_data_smithi138_477915_object186:head# 2024-09-15T05:47:52.533 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.567 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c836a7f2:::benchmark_data_smithi138_477915_object392:head# 2024-09-15T05:47:52.567 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.600 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c84b5188:::benchmark_data_smithi138_477915_object282:head# 2024-09-15T05:47:52.600 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.633 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8f99db7:::benchmark_data_smithi138_477915_object295:head# 2024-09-15T05:47:52.634 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.692 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cac2fc5a:::benchmark_data_smithi138_477915_object380:head# 2024-09-15T05:47:52.692 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.742 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cb485fde:::benchmark_data_smithi138_477915_object342:head# 2024-09-15T05:47:52.743 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.776 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cbb3fcfe:::benchmark_data_smithi138_477915_object258:head# 2024-09-15T05:47:52.776 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.811 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd46d84a:::benchmark_data_smithi138_477915_object337:head# 2024-09-15T05:47:52.811 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.843 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd852d6c:::benchmark_data_smithi138_477915_object107:head# 2024-09-15T05:47:52.843 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.876 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd8a05cc:::benchmark_data_smithi138_477915_object2:head# 2024-09-15T05:47:52.877 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.910 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cec5b2bb:::benchmark_data_smithi138_477915_object25:head# 2024-09-15T05:47:52.910 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.943 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d0e38665:::benchmark_data_smithi138_477915_object266:head# 2024-09-15T05:47:52.943 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:52.977 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d165ae0f:::benchmark_data_smithi138_477915_object211:head# 2024-09-15T05:47:52.977 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.010 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d20fbe61:::benchmark_data_smithi138_477915_object284:head# 2024-09-15T05:47:53.010 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.044 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d350cfaa:::benchmark_data_smithi138_477915_object113:head# 2024-09-15T05:47:53.044 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.069 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d3b5e940:::benchmark_data_smithi138_477915_object14:head# 2024-09-15T05:47:53.069 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.094 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d48aafda:::benchmark_data_smithi138_477915_object358:head# 2024-09-15T05:47:53.095 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.127 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5a212a5:::benchmark_data_smithi138_477915_object106:head# 2024-09-15T05:47:53.127 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.152 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5d6c19c:::benchmark_data_smithi138_477915_object355:head# 2024-09-15T05:47:53.153 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.211 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6d89b5a:::benchmark_data_smithi138_477915_object341:head# 2024-09-15T05:47:53.211 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.262 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d789c5f7:::benchmark_data_smithi138_477915_object336:head# 2024-09-15T05:47:53.262 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.337 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c4ab31:::benchmark_data_smithi138_477915_object189:head# 2024-09-15T05:47:53.337 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.362 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c508f3:::benchmark_data_smithi138_477915_object4:head# 2024-09-15T05:47:53.362 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.387 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7ebcc2e:::benchmark_data_smithi138_477915_object265:head# 2024-09-15T05:47:53.387 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.412 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8daf539:::benchmark_data_smithi138_477915_object99:head# 2024-09-15T05:47:53.413 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8e85bad:::benchmark_data_smithi138_477915_object166:head# 2024-09-15T05:47:53.438 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.471 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d95dedd9:::benchmark_data_smithi138_477915_object274:head# 2024-09-15T05:47:53.471 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.504 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da2f71a7:::benchmark_data_smithi138_477915_object65:head# 2024-09-15T05:47:53.505 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.538 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da407726:::benchmark_data_smithi138_477915_object179:head# 2024-09-15T05:47:53.538 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.572 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dabfd9f7:::benchmark_data_smithi138_477915_object118:head# 2024-09-15T05:47:53.572 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.605 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dad5a76a:::benchmark_data_smithi138_477915_object305:head# 2024-09-15T05:47:53.605 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.639 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6aea31:::benchmark_data_smithi138_477915_object150:head# 2024-09-15T05:47:53.639 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.672 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd78a891:::benchmark_data_smithi138_477915_object353:head# 2024-09-15T05:47:53.672 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.705 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfac43cc:::benchmark_data_smithi138_477915_object126:head# 2024-09-15T05:47:53.706 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.739 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfe8aa68:::benchmark_data_smithi138_477915_object51:head# 2024-09-15T05:47:53.739 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.773 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e100c584:::benchmark_data_smithi138_477915_object8:head# 2024-09-15T05:47:53.773 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.806 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1828df5:::benchmark_data_smithi138_477915_object267:head# 2024-09-15T05:47:53.806 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.840 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1daa4ae:::benchmark_data_smithi138_477915_object37:head# 2024-09-15T05:47:53.840 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.898 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e23cd4da:::benchmark_data_smithi138_477915_object32:head# 2024-09-15T05:47:53.898 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.932 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4da1a27:::benchmark_data_smithi138_477915_object15:head# 2024-09-15T05:47:53.932 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.965 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4f1a68f:::benchmark_data_smithi138_477915_object395:head# 2024-09-15T05:47:53.965 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:53.998 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e6fdeca3:::benchmark_data_smithi138_477915_object338:head# 2024-09-15T05:47:53.999 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.033 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7c862ab:::benchmark_data_smithi138_477915_object119:head# 2024-09-15T05:47:54.033 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.066 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7e5da90:::benchmark_data_smithi138_477915_object393:head# 2024-09-15T05:47:54.066 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.099 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e9e1dfd6:::benchmark_data_smithi138_477915_object315:head# 2024-09-15T05:47:54.099 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.133 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eaee7ade:::benchmark_data_smithi138_477915_object277:head# 2024-09-15T05:47:54.133 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.166 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb2d0c81:::benchmark_data_smithi138_477915_object352:head# 2024-09-15T05:47:54.166 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.200 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb74e80f:::benchmark_data_smithi138_477915_object39:head# 2024-09-15T05:47:54.200 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.233 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ebd302e1:::benchmark_data_smithi138_477915_object221:head# 2024-09-15T05:47:54.233 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.266 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec96583a:::benchmark_data_smithi138_477915_object6:head# 2024-09-15T05:47:54.267 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.308 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecd13f7c:::benchmark_data_smithi138_477915_object174:head# 2024-09-15T05:47:54.308 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.350 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecda3af5:::benchmark_data_smithi138_477915_object7:head# 2024-09-15T05:47:54.350 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.383 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed2124e8:::benchmark_data_smithi138_477915_object283:head# 2024-09-15T05:47:54.384 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.417 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed4f8904:::benchmark_data_smithi138_477915_object321:head# 2024-09-15T05:47:54.417 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.451 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eec767c6:::benchmark_data_smithi138_477915_object304:head# 2024-09-15T05:47:54.451 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.484 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef20ff5b:::benchmark_data_smithi138_477915_object148:head# 2024-09-15T05:47:54.484 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.518 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef304116:::benchmark_data_smithi138_477915_object49:head# 2024-09-15T05:47:54.518 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.551 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef573daf:::benchmark_data_smithi138_477915_object177:head# 2024-09-15T05:47:54.551 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.584 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f0690f60:::benchmark_data_smithi138_477915_object397:head# 2024-09-15T05:47:54.585 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.618 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f15d302a:::benchmark_data_smithi138_477915_object81:head# 2024-09-15T05:47:54.618 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.660 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f2b2bc3f:::benchmark_data_smithi138_477915_object216:head# 2024-09-15T05:47:54.660 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.718 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f314e879:::benchmark_data_smithi138_477915_object290:head# 2024-09-15T05:47:54.719 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.769 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f358deca:::benchmark_data_smithi138_477915_object278:head# 2024-09-15T05:47:54.769 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.802 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f40ce7f6:::benchmark_data_smithi138_477915_object9:head# 2024-09-15T05:47:54.802 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.836 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f48dff42:::benchmark_data_smithi138_477915_object372:head# 2024-09-15T05:47:54.836 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.869 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4a59f6a:::benchmark_data_smithi138_477915_object12:head# 2024-09-15T05:47:54.869 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.903 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4d52c7d:::benchmark_data_smithi138_477915_object183:head# 2024-09-15T05:47:54.903 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.936 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f50e2ea1:::benchmark_data_smithi138_477915_object194:head# 2024-09-15T05:47:54.936 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.961 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5165677:::benchmark_data_smithi138_477915_object346:head# 2024-09-15T05:47:54.961 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:54.987 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5ba49b3:::benchmark_data_smithi138_477915_object333:head# 2024-09-15T05:47:54.987 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.011 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5c42dd1:::benchmark_data_smithi138_477915_object85:head# 2024-09-15T05:47:55.011 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.037 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f64f6100:::benchmark_data_smithi138_477915_object163:head# 2024-09-15T05:47:55.037 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.079 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f6daa659:::benchmark_data_smithi138_477915_object250:head# 2024-09-15T05:47:55.079 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.103 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f7f49d03:::benchmark_data_smithi138_477915_object324:head# 2024-09-15T05:47:55.104 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.129 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa26fae3:::benchmark_data_smithi138_477915_object73:head# 2024-09-15T05:47:55.129 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.154 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa6cc2ae:::benchmark_data_smithi138_477915_object91:head# 2024-09-15T05:47:55.154 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.204 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa9ab79f:::benchmark_data_smithi138_477915_object72:head# 2024-09-15T05:47:55.204 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.255 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fb6a8c6a:::benchmark_data_smithi138_477915_object122:head# 2024-09-15T05:47:55.255 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.322 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fba968a6:::benchmark_data_smithi138_477915_object368:head# 2024-09-15T05:47:55.322 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.347 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fd402ec6:::benchmark_data_smithi138_477915_object67:head# 2024-09-15T05:47:55.347 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.372 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fdef6d22:::benchmark_data_smithi138_477915_object309:head# 2024-09-15T05:47:55.372 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.397 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fe273b33:::benchmark_data_smithi138_477915_object146:head# 2024-09-15T05:47:55.397 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fea7c25b:::benchmark_data_smithi138_477915_object344:head# 2024-09-15T05:47:55.422 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.448 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ff1959bb:::benchmark_data_smithi138_477915_object80:head# 2024-09-15T05:47:55.448 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.481 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffb3fc0b:::benchmark_data_smithi138_477915_object115:head# 2024-09-15T05:47:55.481 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.515 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffbe43c0:::benchmark_data_smithi138_477915_object125:head# 2024-09-15T05:47:55.515 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:55.548 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 9 info 1.1( v 9'194 (0'0,9'194] local-lis/les=7/8 n=194 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:47:55.598 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:47:56.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-15T05:47:57.429 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-09-15T05:47:57.429 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:47:57.429 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 2 -> 1 2024-09-15T05:47:57.429 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:004eb6a1:::benchmark_data_smithi138_477915_object24:head# 2024-09-15T05:47:57.429 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.512 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:01723c07:::benchmark_data_smithi138_477915_object335:head# 2024-09-15T05:47:57.512 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.562 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:021e9c3c:::benchmark_data_smithi138_477915_object60:head# 2024-09-15T05:47:57.563 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.613 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0276aebd:::benchmark_data_smithi138_477915_object108:head# 2024-09-15T05:47:57.613 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.680 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03123a40:::benchmark_data_smithi138_477915_object176:head# 2024-09-15T05:47:57.680 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.747 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:031b6711:::benchmark_data_smithi138_477915_object198:head# 2024-09-15T05:47:57.747 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.789 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0432e19c:::benchmark_data_smithi138_477915_object62:head# 2024-09-15T05:47:57.789 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.839 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04a09b67:::benchmark_data_smithi138_477915_object401:head# 2024-09-15T05:47:57.839 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.906 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04af6a20:::benchmark_data_smithi138_477915_object98:head# 2024-09-15T05:47:57.906 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:57.957 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05c5c04d:::benchmark_data_smithi138_477915_object195:head# 2024-09-15T05:47:57.957 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.007 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05ce29b1:::benchmark_data_smithi138_477915_object181:head# 2024-09-15T05:47:58.007 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.057 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05d45069:::benchmark_data_smithi138_477915_object275:head# 2024-09-15T05:47:58.057 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.107 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07351a71:::benchmark_data_smithi138_477915_object100:head# 2024-09-15T05:47:58.107 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.158 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07fbc7df:::benchmark_data_smithi138_477915_object312:head# 2024-09-15T05:47:58.158 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.208 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08079dbc:::benchmark_data_smithi138_477915_object299:head# 2024-09-15T05:47:58.208 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.267 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08859349:::benchmark_data_smithi138_477915_object223:head# 2024-09-15T05:47:58.267 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.317 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08a0ce86:::benchmark_data_smithi138_477915_object385:head# 2024-09-15T05:47:58.317 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.342 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08baf2ed:::benchmark_data_smithi138_477915_object192:head# 2024-09-15T05:47:58.342 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.367 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08bd9a02:::benchmark_data_smithi138_477915_object214:head# 2024-09-15T05:47:58.367 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.401 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:097fa44d:::benchmark_data_smithi138_477915_object328:head# 2024-09-15T05:47:58.401 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.434 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a0d4136:::benchmark_data_smithi138_477915_object30:head# 2024-09-15T05:47:58.434 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.468 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2641dd:::benchmark_data_smithi138_477915_object210:head# 2024-09-15T05:47:58.468 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.501 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2d6d17:::benchmark_data_smithi138_477915_object61:head# 2024-09-15T05:47:58.501 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.534 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ac84968:::benchmark_data_smithi138_477915_object286:head# 2024-09-15T05:47:58.534 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.568 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ad209ab:::benchmark_data_smithi138_477915_object70:head# 2024-09-15T05:47:58.568 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.601 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0b64715f:::benchmark_data_smithi138_477915_object17:head# 2024-09-15T05:47:58.602 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.635 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0bbf6561:::benchmark_data_smithi138_477915_object137:head# 2024-09-15T05:47:58.635 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.702 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0cca7b5b:::benchmark_data_smithi138_477915_object339:head# 2024-09-15T05:47:58.702 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.752 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0d194a0a:::benchmark_data_smithi138_477915_object77:head# 2024-09-15T05:47:58.752 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.786 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e135564:::benchmark_data_smithi138_477915_object313:head# 2024-09-15T05:47:58.786 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.819 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e15c428:::benchmark_data_smithi138_477915_object120:head# 2024-09-15T05:47:58.819 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.853 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0f4410e4:::benchmark_data_smithi138_477915_object145:head# 2024-09-15T05:47:58.853 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.886 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0fefd637:::benchmark_data_smithi138_477915_object389:head# 2024-09-15T05:47:58.886 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.920 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10535269:::benchmark_data_smithi138_477915_object332:head# 2024-09-15T05:47:58.920 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.953 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10bc54c8:::benchmark_data_smithi138_477915_object225:head# 2024-09-15T05:47:58.953 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:58.986 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10e6b441:::benchmark_data_smithi138_477915_object248:head# 2024-09-15T05:47:58.986 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.045 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1161a4f5:::benchmark_data_smithi138_477915_object281:head# 2024-09-15T05:47:59.045 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.078 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11ab1fa3:::benchmark_data_smithi138_477915_object240:head# 2024-09-15T05:47:59.079 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.112 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11b92b78:::benchmark_data_smithi138_477915_object142:head# 2024-09-15T05:47:59.112 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.145 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:13efdddb:::benchmark_data_smithi138_477915_object57:head# 2024-09-15T05:47:59.145 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.183 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14c54418:::benchmark_data_smithi138_477915_object197:head# 2024-09-15T05:47:59.184 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.229 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:152a39dd:::benchmark_data_smithi138_477915_object144:head# 2024-09-15T05:47:59.229 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.280 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15322a7a:::benchmark_data_smithi138_477915_object288:head# 2024-09-15T05:47:59.281 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.347 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15aaf075:::benchmark_data_smithi138_477915_object128:head# 2024-09-15T05:47:59.347 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.372 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16dad5c5:::benchmark_data_smithi138_477915_object252:head# 2024-09-15T05:47:59.372 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.397 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16f17ac6:::benchmark_data_smithi138_477915_object329:head# 2024-09-15T05:47:59.397 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1708f8e0:::benchmark_data_smithi138_477915_object307:head# 2024-09-15T05:47:59.422 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.448 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1854249c:::benchmark_data_smithi138_477915_object46:head# 2024-09-15T05:47:59.448 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.473 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:191dd67b:::benchmark_data_smithi138_477915_object20:head# 2024-09-15T05:47:59.473 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.498 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1acb70bd:::benchmark_data_smithi138_477915_object19:head# 2024-09-15T05:47:59.498 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.523 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b1b6402:::benchmark_data_smithi138_477915_object196:head# 2024-09-15T05:47:59.523 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.548 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b7cd855:::benchmark_data_smithi138_477915_object243:head# 2024-09-15T05:47:59.548 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.573 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1baea71e:::benchmark_data_smithi138_477915_object330:head# 2024-09-15T05:47:59.574 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.598 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1c0c4030:::benchmark_data_smithi138_477915_object325:head# 2024-09-15T05:47:59.599 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.624 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1ced30b9:::benchmark_data_smithi138_477915_object165:head# 2024-09-15T05:47:59.624 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.649 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dac8c2e:::benchmark_data_smithi138_477915_object204:head# 2024-09-15T05:47:59.649 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.674 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dc29afc:::benchmark_data_smithi138_477915_object398:head# 2024-09-15T05:47:59.674 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.699 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1f7be81b:::benchmark_data_smithi138_477915_object292:head# 2024-09-15T05:47:59.699 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.724 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20000fdb:::benchmark_data_smithi138_477915_object157:head# 2024-09-15T05:47:59.724 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.750 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20ce187a:::benchmark_data_smithi138_477915_object116:head# 2024-09-15T05:47:59.750 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.774 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:211ff0cd:::benchmark_data_smithi138_477915_object209:head# 2024-09-15T05:47:59.774 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.799 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:214fff42:::benchmark_data_smithi138_477915_object230:head# 2024-09-15T05:47:59.800 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.825 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:219197a2:::benchmark_data_smithi138_477915_object111:head# 2024-09-15T05:47:59.825 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.850 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:22101c95:::benchmark_data_smithi138_477915_object59:head# 2024-09-15T05:47:59.850 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.875 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:223f5228:::benchmark_data_smithi138_477915_object79:head# 2024-09-15T05:47:59.875 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.934 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:230de784:::benchmark_data_smithi138_477915_object244:head# 2024-09-15T05:47:59.934 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.959 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:246b4656:::benchmark_data_smithi138_477915_object191:head# 2024-09-15T05:47:59.959 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:47:59.984 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:25092a54:::benchmark_data_smithi138_477915_object10:head# 2024-09-15T05:47:59.984 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.009 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:263d2c7b:::benchmark_data_smithi138_477915_object153:head# 2024-09-15T05:48:00.010 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.034 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:266b4db5:::benchmark_data_smithi138_477915_object13:head# 2024-09-15T05:48:00.035 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.071 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26e68133:::benchmark_data_smithi138_477915_object349:head# 2024-09-15T05:48:00.072 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.102 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26f80018:::benchmark_data_smithi138_477915_object75:head# 2024-09-15T05:48:00.102 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.126 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2729df23:::benchmark_data_smithi138_477915_object245:head# 2024-09-15T05:48:00.127 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.152 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:273a39c8:::benchmark_data_smithi138_477915_object212:head# 2024-09-15T05:48:00.152 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.185 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b21cb84:::benchmark_data_smithi138_477915_object366:head# 2024-09-15T05:48:00.185 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.219 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2e0a6ba0:::benchmark_data_smithi138_477915_object45:head# 2024-09-15T05:48:00.219 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.252 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2fd3771a:::benchmark_data_smithi138_477915_object326:head# 2024-09-15T05:48:00.252 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.285 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30de888f:::benchmark_data_smithi138_477915_object34:head# 2024-09-15T05:48:00.286 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.319 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:312dc4e7:::benchmark_data_smithi138_477915_object232:head# 2024-09-15T05:48:00.319 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.353 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:318ba285:::benchmark_data_smithi138_477915_object164:head# 2024-09-15T05:48:00.353 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.386 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:31e1d00e:::benchmark_data_smithi138_477915_object44:head# 2024-09-15T05:48:00.386 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.419 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:32a36681:::benchmark_data_smithi138_477915_object90:head# 2024-09-15T05:48:00.420 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.453 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3397aa63:::benchmark_data_smithi138_477915_object294:head# 2024-09-15T05:48:00.453 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.486 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:350297ef:::benchmark_data_smithi138_477915_object382:head# 2024-09-15T05:48:00.486 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.528 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:354f77ae:::benchmark_data_smithi138_477915_object378:head# 2024-09-15T05:48:00.528 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.562 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:359485cb:::benchmark_data_smithi138_477915_object363:head# 2024-09-15T05:48:00.562 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.595 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:386dd901:::benchmark_data_smithi138_477915_object287:head# 2024-09-15T05:48:00.595 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.629 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:389afc62:::benchmark_data_smithi138_477915_object206:head# 2024-09-15T05:48:00.629 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.666 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38edce5f:::benchmark_data_smithi138_477915_object202:head# 2024-09-15T05:48:00.667 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.721 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:392023cd:::benchmark_data_smithi138_477915_object360:head# 2024-09-15T05:48:00.721 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.771 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:396a3af9:::benchmark_data_smithi138_477915_object361:head# 2024-09-15T05:48:00.771 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.805 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39870a18:::benchmark_data_smithi138_477915_object55:head# 2024-09-15T05:48:00.805 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.838 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a34f987:::benchmark_data_smithi138_477915_object84:head# 2024-09-15T05:48:00.838 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.872 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a45d3f0:::benchmark_data_smithi138_477915_object133:head# 2024-09-15T05:48:00.872 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.905 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc519ab:::benchmark_data_smithi138_477915_object301:head# 2024-09-15T05:48:00.905 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.930 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc57672:::benchmark_data_smithi138_477915_object317:head# 2024-09-15T05:48:00.930 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.955 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d290d02:::benchmark_data_smithi138_477915_object52:head# 2024-09-15T05:48:00.955 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:00.981 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d995ed5:::benchmark_data_smithi138_477915_object289:head# 2024-09-15T05:48:00.981 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.006 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3e073204:::benchmark_data_smithi138_477915_object47:head# 2024-09-15T05:48:01.006 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.039 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f24c379:::benchmark_data_smithi138_477915_object381:head# 2024-09-15T05:48:01.040 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.065 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f8ef9b0:::benchmark_data_smithi138_477915_object215:head# 2024-09-15T05:48:01.065 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.090 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4188c502:::benchmark_data_smithi138_477915_object394:head# 2024-09-15T05:48:01.090 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.115 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:41d0e70f:::benchmark_data_smithi138_477915_object124:head# 2024-09-15T05:48:01.115 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.140 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:42ce4ca2:::benchmark_data_smithi138_477915_object170:head# 2024-09-15T05:48:01.140 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.165 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4384dd5f:::benchmark_data_smithi138_477915_object222:head# 2024-09-15T05:48:01.166 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.232 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:441e81fc:::benchmark_data_smithi138_477915_object322:head# 2024-09-15T05:48:01.232 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.291 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:444f377d:::benchmark_data_smithi138_477915_object259:head# 2024-09-15T05:48:01.291 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.341 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44d1d1a6:::benchmark_data_smithi138_477915_object175:head# 2024-09-15T05:48:01.341 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.366 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44e95b23:::benchmark_data_smithi138_477915_object187:head# 2024-09-15T05:48:01.366 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.391 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44f216b2:::benchmark_data_smithi138_477915_object331:head# 2024-09-15T05:48:01.392 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.417 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45f59372:::benchmark_data_smithi138_477915_object134:head# 2024-09-15T05:48:01.417 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.442 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:463b99f0:::benchmark_data_smithi138_477915_object41:head# 2024-09-15T05:48:01.442 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.467 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:46f42845:::benchmark_data_smithi138_477915_object50:head# 2024-09-15T05:48:01.467 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.492 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:47847711:::benchmark_data_smithi138_477915_object391:head# 2024-09-15T05:48:01.492 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.517 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:481a4dcf:::benchmark_data_smithi138_477915_object306:head# 2024-09-15T05:48:01.517 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.584 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48339ebf:::benchmark_data_smithi138_477915_object53:head# 2024-09-15T05:48:01.584 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.609 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48cd125a:::benchmark_data_smithi138_477915_object320:head# 2024-09-15T05:48:01.609 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.643 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:49297529:::benchmark_data_smithi138_477915_object110:head# 2024-09-15T05:48:01.643 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.701 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:492bb000:::benchmark_data_smithi138_477915_object364:head# 2024-09-15T05:48:01.702 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.752 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4a0a3214:::benchmark_data_smithi138_477915_object220:head# 2024-09-15T05:48:01.752 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.810 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4b9543c5:::benchmark_data_smithi138_477915_object1:head# 2024-09-15T05:48:01.810 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.852 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba0927c:::benchmark_data_smithi138_477915_object178:head# 2024-09-15T05:48:01.852 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.919 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba2f324:::benchmark_data_smithi138_477915_object293:head# 2024-09-15T05:48:01.919 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.953 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c0b6957:::benchmark_data_smithi138_477915_object114:head# 2024-09-15T05:48:01.953 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:01.986 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c7779cc:::benchmark_data_smithi138_477915_object379:head# 2024-09-15T05:48:01.986 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.020 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4cffde3a:::benchmark_data_smithi138_477915_object271:head# 2024-09-15T05:48:02.020 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.053 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d3925ed:::benchmark_data_smithi138_477915_object396:head# 2024-09-15T05:48:02.053 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.087 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d67e2bb:::benchmark_data_smithi138_477915_object141:head# 2024-09-15T05:48:02.087 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.120 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4e93ce33:::benchmark_data_smithi138_477915_object82:head# 2024-09-15T05:48:02.120 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.187 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4eb39cc3:::benchmark_data_smithi138_477915_object11:head# 2024-09-15T05:48:02.187 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.221 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4fdb5ea1:::benchmark_data_smithi138_477915_object86:head# 2024-09-15T05:48:02.221 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.254 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52b28e51:::benchmark_data_smithi138_477915_object188:head# 2024-09-15T05:48:02.254 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.288 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52f1d98b:::benchmark_data_smithi138_477915_object29:head# 2024-09-15T05:48:02.288 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.321 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:538d02e4:::benchmark_data_smithi138_477915_object387:head# 2024-09-15T05:48:02.321 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.355 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:53b28f04:::benchmark_data_smithi138_477915_object193:head# 2024-09-15T05:48:02.355 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.388 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:543e73a1:::benchmark_data_smithi138_477915_object143:head# 2024-09-15T05:48:02.388 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:546413ea:::benchmark_data_smithi138_477915_object35:head# 2024-09-15T05:48:02.422 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.472 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54b5d110:::benchmark_data_smithi138_477915_object370:head# 2024-09-15T05:48:02.472 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.505 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54d808a1:::benchmark_data_smithi138_477915_object147:head# 2024-09-15T05:48:02.505 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.539 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:556f515c:::benchmark_data_smithi138_477915_object367:head# 2024-09-15T05:48:02.539 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.572 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5581a3b9:::benchmark_data_smithi138_477915_object123:head# 2024-09-15T05:48:02.572 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.605 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55b4ce8c:::benchmark_data_smithi138_477915_object327:head# 2024-09-15T05:48:02.606 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.641 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55e0f391:::benchmark_data_smithi138_477915_object40:head# 2024-09-15T05:48:02.642 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.692 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:564dfa9a:::benchmark_data_smithi138_477915_object89:head# 2024-09-15T05:48:02.692 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.750 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56619472:::benchmark_data_smithi138_477915_object291:head# 2024-09-15T05:48:02.750 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.775 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56ac2e18:::benchmark_data_smithi138_477915_object373:head# 2024-09-15T05:48:02.775 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.800 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e0ff0d:::benchmark_data_smithi138_477915_object246:head# 2024-09-15T05:48:02.800 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.825 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e54ad7:::benchmark_data_smithi138_477915_object241:head# 2024-09-15T05:48:02.825 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.851 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:572a26cc:::benchmark_data_smithi138_477915_object3:head# 2024-09-15T05:48:02.851 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.876 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:57365c69:::benchmark_data_smithi138_477915_object185:head# 2024-09-15T05:48:02.876 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.901 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:58b05a98:::benchmark_data_smithi138_477915_object94:head# 2024-09-15T05:48:02.901 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.926 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5979f860:::benchmark_data_smithi138_477915_object42:head# 2024-09-15T05:48:02.927 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.951 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5a3930c3:::benchmark_data_smithi138_477915_object92:head# 2024-09-15T05:48:02.951 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:02.977 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5acdf762:::benchmark_data_smithi138_477915_object217:head# 2024-09-15T05:48:02.977 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.001 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5b192008:::benchmark_data_smithi138_477915_object190:head# 2024-09-15T05:48:03.002 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.026 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5bd647c9:::benchmark_data_smithi138_477915_object308:head# 2024-09-15T05:48:03.027 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.052 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c5abf37:::benchmark_data_smithi138_477915_object227:head# 2024-09-15T05:48:03.052 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.077 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5cc75e94:::benchmark_data_smithi138_477915_object261:head# 2024-09-15T05:48:03.077 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.110 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5d452870:::benchmark_data_smithi138_477915_object386:head# 2024-09-15T05:48:03.110 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.144 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5e2e9211:::benchmark_data_smithi138_477915_object201:head# 2024-09-15T05:48:03.144 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.169 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5f10ef7e:::benchmark_data_smithi138_477915_object182:head# 2024-09-15T05:48:03.169 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.219 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fa8bc7a:::benchmark_data_smithi138_477915_object371:head# 2024-09-15T05:48:03.220 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.287 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fd3194b:::benchmark_data_smithi138_477915_object83:head# 2024-09-15T05:48:03.287 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.328 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fec7540:::benchmark_data_smithi138_477915_object132:head# 2024-09-15T05:48:03.329 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.354 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6012bbb1:::benchmark_data_smithi138_477915_object213:head# 2024-09-15T05:48:03.354 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.379 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:620660f2:::benchmark_data_smithi138_477915_object138:head# 2024-09-15T05:48:03.379 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.412 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:62fcea3a:::benchmark_data_smithi138_477915_object374:head# 2024-09-15T05:48:03.412 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:635fcc02:::benchmark_data_smithi138_477915_object18:head# 2024-09-15T05:48:03.437 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.471 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6429912b:::benchmark_data_smithi138_477915_object78:head# 2024-09-15T05:48:03.471 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.504 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:648cc3c2:::benchmark_data_smithi138_477915_object180:head# 2024-09-15T05:48:03.504 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.538 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64ae8f93:::benchmark_data_smithi138_477915_object273:head# 2024-09-15T05:48:03.538 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.571 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:65851d9e:::benchmark_data_smithi138_477915_object276:head# 2024-09-15T05:48:03.571 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.605 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:66acc5ee:::benchmark_data_smithi138_477915_object155:head# 2024-09-15T05:48:03.605 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.638 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:67e15da6:::benchmark_data_smithi138_477915_object253:head# 2024-09-15T05:48:03.638 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.672 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68a5ecca:::benchmark_data_smithi138_477915_object260:head# 2024-09-15T05:48:03.672 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.705 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68b33428:::benchmark_data_smithi138_477915_object388:head# 2024-09-15T05:48:03.705 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.739 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6940b9d3:::benchmark_data_smithi138_477915_object239:head# 2024-09-15T05:48:03.739 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.772 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:69f7ef4c:::benchmark_data_smithi138_477915_object127:head# 2024-09-15T05:48:03.773 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6aef21a2:::benchmark_data_smithi138_477915_object54:head# 2024-09-15T05:48:03.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.848 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6b3c2b63:::benchmark_data_smithi138_477915_object208:head# 2024-09-15T05:48:03.848 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.881 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bf4848f:::benchmark_data_smithi138_477915_object399:head# 2024-09-15T05:48:03.881 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.914 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ccf34bf:::benchmark_data_smithi138_477915_object22:head# 2024-09-15T05:48:03.914 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.948 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6d5a3232:::benchmark_data_smithi138_477915_object43:head# 2024-09-15T05:48:03.948 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:03.981 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dc7f41b:::benchmark_data_smithi138_477915_object262:head# 2024-09-15T05:48:03.981 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.015 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dd17fb9:::benchmark_data_smithi138_477915_object263:head# 2024-09-15T05:48:04.015 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.048 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ddcb643:::benchmark_data_smithi138_477915_object285:head# 2024-09-15T05:48:04.049 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.082 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6e1716aa:::benchmark_data_smithi138_477915_object135:head# 2024-09-15T05:48:04.082 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.115 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:702ed571:::benchmark_data_smithi138_477915_object219:head# 2024-09-15T05:48:04.115 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.140 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:707bb828:::benchmark_data_smithi138_477915_object158:head# 2024-09-15T05:48:04.140 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.174 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:70ca4348:::benchmark_data_smithi138_477915_object26:head# 2024-09-15T05:48:04.174 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.199 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71784a9a:::benchmark_data_smithi138_477915_object376:head# 2024-09-15T05:48:04.199 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.233 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:717dd36a:::benchmark_data_smithi138_477915_object139:head# 2024-09-15T05:48:04.233 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.258 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71fb5440:::benchmark_data_smithi138_477915_object97:head# 2024-09-15T05:48:04.258 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.325 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7314c8d4:::benchmark_data_smithi138_477915_object152:head# 2024-09-15T05:48:04.325 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.358 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:740159d0:::benchmark_data_smithi138_477915_object0:head# 2024-09-15T05:48:04.358 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.392 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:74886e53:::benchmark_data_smithi138_477915_object112:head# 2024-09-15T05:48:04.392 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.425 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:75046150:::benchmark_data_smithi138_477915_object279:head# 2024-09-15T05:48:04.426 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:773e28fe:::benchmark_data_smithi138_477915_object377:head# 2024-09-15T05:48:04.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.492 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7783b60c:::benchmark_data_smithi138_477915_object101:head# 2024-09-15T05:48:04.493 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.526 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:786160e4:::benchmark_data_smithi138_477915_object38:head# 2024-09-15T05:48:04.526 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.559 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a2bef6b:::benchmark_data_smithi138_477915_object58:head# 2024-09-15T05:48:04.560 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.593 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a76feec:::benchmark_data_smithi138_477915_object63:head# 2024-09-15T05:48:04.593 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.626 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7bb84b6a:::benchmark_data_smithi138_477915_object5:head# 2024-09-15T05:48:04.626 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.668 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7be49943:::benchmark_data_smithi138_477915_object48:head# 2024-09-15T05:48:04.668 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.727 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d976229:::benchmark_data_smithi138_477915_object207:head# 2024-09-15T05:48:04.727 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.777 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d9d7a5f:::benchmark_data_smithi138_477915_object369:head# 2024-09-15T05:48:04.777 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.811 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eaec87b:::benchmark_data_smithi138_477915_object154:head# 2024-09-15T05:48:04.812 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.844 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eda30d0:::benchmark_data_smithi138_477915_object310:head# 2024-09-15T05:48:04.844 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.877 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7f744066:::benchmark_data_smithi138_477915_object205:head# 2024-09-15T05:48:04.878 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:48:04.911 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 9 info 1.0( v 9'209 (0'0,9'209] local-lis/les=7/8 n=209 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:48:04.944 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:48:05.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:48:05.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:48:05.465 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:48:05.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:48:05.467 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:48:05.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:48:05.491 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:48:05.489+0000 7f7f4232b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:48:05.493 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:48:05.491+0000 7f7f4232b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:48:05.494 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:48:05.492+0000 7f7f4232b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:48:05.827 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:48:05.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:48:05.828 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:48:05.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:48:05.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:48:06.012 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:48:06.010+0000 7f7f4232b440 -1 Falling back to public interface 2024-09-15T05:48:06.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:48:07.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:48:07.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:48:07.180 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:48:07.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:48:07.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:48:07.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:48:07.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:48:08.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:48:08.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:48:08.541 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:48:08.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:48:08.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:48:08.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:48:08.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:48:09.018 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:48:09.016+0000 7f7f4232b440 -1 osd.0 20 log_to_monitors true 2024-09-15T05:48:09.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:48:09.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:48:09.901 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:48:09.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:48:09.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:48:09.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:48:10.259 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 21 last_clean_interval [12,20) [v2:127.0.0.1:6802/480703,v1:127.0.0.1:6803/480703] [v2:127.0.0.1:6804/480703,v1:127.0.0.1:6805/480703] exists,up be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:48:10.261 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:48:10.262 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:48:10.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:48:10.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:48:10.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:48:10.346 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:48:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:48:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:48:10.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:48:10.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247811 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247811 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247811' 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247811 2024-09-15T05:48:10.801 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:48:10.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:48:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247811 2024-09-15T05:48:10.802 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:48:10.803 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 98784247811 2024-09-15T05:48:10.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247811 2024-09-15T05:48:10.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247811' 2024-09-15T05:48:10.804 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:48:11.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 98784247811 2024-09-15T05:48:11.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:48:12.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:48:12.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:48:12.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 98784247811 2024-09-15T05:48:12.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:48:13.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-15T05:48:13.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:48:13.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-15T05:48:13.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:48:13.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:48:13.873 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:48:14.343 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:48:14.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:48:14.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:48:14.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:48:14.685 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:48:15.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:48:15.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:48:15.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:48:15.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-09-15T05:48:15.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:15.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:15.801 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:15.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:16.572 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:16.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:17.233 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:17.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:17.970 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:17.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:17.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:18.791 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:18.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:19.485 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:19.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:19.496 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:20.189 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:20.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:21.059 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:21.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:21.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:21.864 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:21.875 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:22.526 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:22.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:22.537 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:23.288 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:23.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:24.034 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:24.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:24.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:24.863 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:24.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:25.607 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:25.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:26.395 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:26.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:27.232 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:27.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:27.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:27.936 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:27.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:28.673 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:28.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:28.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:29.335 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:29.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:30.080 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:30.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:30.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:30.901 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:30.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:31.596 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:31.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:31.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:32.341 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:32.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:33.171 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:33.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:33.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:33.925 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:33.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:34.662 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:34.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:34.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:35.407 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:35.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:36.195 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:36.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:36.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:36.856 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:36.867 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:37.501 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:37.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:37.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:38.213 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:38.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:38.950 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:38.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:38.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:39.687 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:39.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:40.483 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:40.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:40.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:41.253 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:41.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:42.092 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:42.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:42.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:42.879 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:42.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:43.548 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:43.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:43.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:44.310 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:44.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:45.148 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:45.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:45.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:45.885 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:45.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:46.631 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:46.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:46.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:47.334 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:47.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:48.138 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:48.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:48.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:48.968 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:48.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:49.679 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:49.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:49.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:50.483 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:50.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:51.336 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:51.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:51.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:51.998 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:52.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:52.845 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:52.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:52.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:53.572 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:53.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:54.351 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:54.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:54.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:55.096 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:55.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:55.867 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:55.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:56.788 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:56.799 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:57.450 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:57.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:57.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:58.253 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:58.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:48:58.998 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:48:59.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:48:59.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:48:59.777 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:48:59.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:00.531 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:00.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:00.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:01.310 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:01.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:02.301 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:02.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:02.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:02.992 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:03.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:03.629 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:03.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:03.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:04.374 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:04.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:05.079 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:05.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:05.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:05.824 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:05.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:06.568 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:06.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:06.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:07.337 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:07.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:08.090 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:08.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:08.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:08.809 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:08.820 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:09.495 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:09.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:09.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:10.206 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:10.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:11.009 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:11.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:11.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:11.855 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:11.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:12.556 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:12.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:12.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:13.335 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:13.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:14.121 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:14.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:14.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:14.890 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:14.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:15.618 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:15.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:15.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:16.404 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:16.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:17.240 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:17.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:17.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:18.010 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:18.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:18.797 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:18.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:18.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:19.566 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:19.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:20.320 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:20.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:20.331 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:21.056 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:21.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:22.024 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:22.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:22.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:22.855 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:22.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:23.516 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:23.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:23.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:24.227 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:24.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:24.980 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:24.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:24.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:25.709 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:25.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:26.437 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:26.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:26.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:27.383 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:27.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:28.169 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:28.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:28.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:28.939 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:28.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:29.609 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:29.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-09-15T05:49:29.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:30.320 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:30.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:31.048 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:31.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-09-15T05:49:31.086 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T05:49:31.086 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-09-15T05:49:31.086 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_483817 2024-09-15T05:49:31.086 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T05:49:31.086 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T05:49:32.086 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 177 161 0.157211 0.157227 0.100602 0.0896116 2024-09-15T05:49:33.086 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 1.08277 2024-09-15T05:49:33.086 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 177 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 1024 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 1024 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.159638 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.157227 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.157227 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 163 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 0 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 161 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 161 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.0929207 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0273339 2024-09-15T05:49:33.087 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.142506 2024-09-15T05:49:33.088 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0614654 2024-09-15T05:49:33.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-15T05:49:33.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:49:33.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:49:33.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:49:33.123 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:49:33.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:49:33.123 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:49:33.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:49:33.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:49:33.124 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:49:33.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:49:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:49:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:49:33.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:49:33.591 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:49:33.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247829 2024-09-15T05:49:33.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247829 2024-09-15T05:49:33.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-98784247829' 2024-09-15T05:49:33.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:49:33.698 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-98784247829 2024-09-15T05:49:33.699 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:49:33.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:49:33.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-98784247829 2024-09-15T05:49:33.700 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:49:33.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247829 2024-09-15T05:49:33.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 98784247829' 2024-09-15T05:49:33.701 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 98784247829 2024-09-15T05:49:33.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:49:34.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247827 -lt 98784247829 2024-09-15T05:49:34.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:49:35.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:49:35.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:49:35.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247830 -lt 98784247829 2024-09-15T05:49:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:49:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:49:35.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:49:35.981 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:49:35.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:49:35.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:49:36.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:49:36.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:49:36.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:49:36.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:49:36.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:49:36.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:49:36.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:49:36.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-09-15T05:49:36.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-09-15T05:49:41.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-09-15T05:49:42.725 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is set 2024-09-15T05:49:42.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-09-15T05:49:43.386 INFO:tasks.workunit.client.0.smithi138.stderr:nodown is unset 2024-09-15T05:49:43.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-09-15T05:49:48.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-09-15T05:49:48.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:49:48.399 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:49:48.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:49:48.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:49:48.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:49:48.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:49:48.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-15T05:49:50.105 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:49:50.105 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.1 2024-09-15T05:49:50.105 INFO:tasks.workunit.client.0.smithi138.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-15T05:49:50.105 INFO:tasks.workunit.client.0.smithi138.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-15T05:49:50.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-09-15T05:49:51.872 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.0 info 1.0( v 224'387 (9'209,224'387] local-lis/les=23/24 n=580 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-15T05:49:51.888 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:004eb6a1:::benchmark_data_smithi138_477915_object24:head# 2024-09-15T05:49:51.896 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:005bff5c:::benchmark_data_smithi138_483817_object135:head# 2024-09-15T05:49:51.897 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0149ad20:::benchmark_data_smithi138_483817_object63:head# 2024-09-15T05:49:51.913 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:01723c07:::benchmark_data_smithi138_477915_object335:head# 2024-09-15T05:49:51.921 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:021e9c3c:::benchmark_data_smithi138_477915_object60:head# 2024-09-15T05:49:51.923 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0255b854:::benchmark_data_smithi138_483817_object75:head# 2024-09-15T05:49:51.938 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0276aebd:::benchmark_data_smithi138_477915_object108:head# 2024-09-15T05:49:51.938 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:03123a40:::benchmark_data_smithi138_477915_object176:head# 2024-09-15T05:49:51.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:031b6711:::benchmark_data_smithi138_477915_object198:head# 2024-09-15T05:49:51.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:03a96464:::benchmark_data_smithi138_483817_object19:head# 2024-09-15T05:49:51.947 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0432e19c:::benchmark_data_smithi138_477915_object62:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0465873b:::benchmark_data_smithi138_483817_object34:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:04a09b67:::benchmark_data_smithi138_477915_object401:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:04af6a20:::benchmark_data_smithi138_477915_object98:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05c5c04d:::benchmark_data_smithi138_477915_object195:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05ce29b1:::benchmark_data_smithi138_477915_object181:head# 2024-09-15T05:49:51.950 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05d45069:::benchmark_data_smithi138_477915_object275:head# 2024-09-15T05:49:51.951 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:07351a71:::benchmark_data_smithi138_477915_object100:head# 2024-09-15T05:49:51.951 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:07fbc7df:::benchmark_data_smithi138_477915_object312:head# 2024-09-15T05:49:51.951 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08079dbc:::benchmark_data_smithi138_477915_object299:head# 2024-09-15T05:49:51.951 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08859349:::benchmark_data_smithi138_477915_object223:head# 2024-09-15T05:49:51.951 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08a0ce86:::benchmark_data_smithi138_477915_object385:head# 2024-09-15T05:49:51.956 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08baf2ed:::benchmark_data_smithi138_477915_object192:head# 2024-09-15T05:49:51.957 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08bd9a02:::benchmark_data_smithi138_477915_object214:head# 2024-09-15T05:49:51.957 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0973fe62:::benchmark_data_smithi138_483817_object92:head# 2024-09-15T05:49:51.957 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:097fa44d:::benchmark_data_smithi138_477915_object328:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:09e06460:::benchmark_data_smithi138_483817_object133:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a0d4136:::benchmark_data_smithi138_477915_object30:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a2641dd:::benchmark_data_smithi138_477915_object210:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a2d6d17:::benchmark_data_smithi138_477915_object61:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ac84968:::benchmark_data_smithi138_477915_object286:head# 2024-09-15T05:49:51.959 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ad209ab:::benchmark_data_smithi138_477915_object70:head# 2024-09-15T05:49:51.960 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0af91a19:::benchmark_data_smithi138_483817_object15:head# 2024-09-15T05:49:51.960 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0b64715f:::benchmark_data_smithi138_477915_object17:head# 2024-09-15T05:49:51.961 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ba493f4:::benchmark_data_smithi138_483817_object86:head# 2024-09-15T05:49:51.962 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0bbf6561:::benchmark_data_smithi138_477915_object137:head# 2024-09-15T05:49:51.962 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0cca7b5b:::benchmark_data_smithi138_477915_object339:head# 2024-09-15T05:49:51.962 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0cd0de56:::benchmark_data_smithi138_483817_object122:head# 2024-09-15T05:49:51.963 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0d194a0a:::benchmark_data_smithi138_477915_object77:head# 2024-09-15T05:49:51.963 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0e09c97e:::benchmark_data_smithi138_483817_object82:head# 2024-09-15T05:49:51.964 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0e135564:::benchmark_data_smithi138_477915_object313:head# 2024-09-15T05:49:51.964 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0e15c428:::benchmark_data_smithi138_477915_object120:head# 2024-09-15T05:49:51.965 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0f4410e4:::benchmark_data_smithi138_477915_object145:head# 2024-09-15T05:49:51.965 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0f9a013a:::benchmark_data_smithi138_483817_object76:head# 2024-09-15T05:49:51.965 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0fefd637:::benchmark_data_smithi138_477915_object389:head# 2024-09-15T05:49:51.966 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10535269:::benchmark_data_smithi138_477915_object332:head# 2024-09-15T05:49:51.966 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10796dab:::benchmark_data_smithi138_483817_object99:head# 2024-09-15T05:49:51.966 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10bc54c8:::benchmark_data_smithi138_477915_object225:head# 2024-09-15T05:49:51.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10e6b441:::benchmark_data_smithi138_477915_object248:head# 2024-09-15T05:49:51.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:111a5576:::benchmark_data_smithi138_483817_object124:head# 2024-09-15T05:49:51.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1161a4f5:::benchmark_data_smithi138_477915_object281:head# 2024-09-15T05:49:51.967 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:11821dac:::benchmark_data_smithi138_483817_object62:head# 2024-09-15T05:49:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:11ab1fa3:::benchmark_data_smithi138_477915_object240:head# 2024-09-15T05:49:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:11b92b78:::benchmark_data_smithi138_477915_object142:head# 2024-09-15T05:49:51.968 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:13efdddb:::benchmark_data_smithi138_477915_object57:head# 2024-09-15T05:49:51.969 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:14c54418:::benchmark_data_smithi138_477915_object197:head# 2024-09-15T05:49:51.969 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:152a39dd:::benchmark_data_smithi138_477915_object144:head# 2024-09-15T05:49:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15322a7a:::benchmark_data_smithi138_477915_object288:head# 2024-09-15T05:49:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15aaf075:::benchmark_data_smithi138_477915_object128:head# 2024-09-15T05:49:51.970 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16a80714:::benchmark_data_smithi138_483817_object56:head# 2024-09-15T05:49:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16d2ffca:::benchmark_data_smithi138_483817_object168:head# 2024-09-15T05:49:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16dad5c5:::benchmark_data_smithi138_477915_object252:head# 2024-09-15T05:49:51.972 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16f17ac6:::benchmark_data_smithi138_477915_object329:head# 2024-09-15T05:49:51.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1708f8e0:::benchmark_data_smithi138_477915_object307:head# 2024-09-15T05:49:51.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1854249c:::benchmark_data_smithi138_477915_object46:head# 2024-09-15T05:49:51.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:18ee0375:::benchmark_data_smithi138_483817_object18:head# 2024-09-15T05:49:51.973 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:19010e40:::benchmark_data_smithi138_483817_object96:head# 2024-09-15T05:49:51.974 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:191dd67b:::benchmark_data_smithi138_477915_object20:head# 2024-09-15T05:49:51.974 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1a10802d:::benchmark_data_smithi138_483817_object11:head# 2024-09-15T05:49:51.975 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1acb70bd:::benchmark_data_smithi138_477915_object19:head# 2024-09-15T05:49:51.975 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b1b6402:::benchmark_data_smithi138_477915_object196:head# 2024-09-15T05:49:51.975 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b49f001:::benchmark_data_smithi138_483817_object77:head# 2024-09-15T05:49:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b7cd855:::benchmark_data_smithi138_477915_object243:head# 2024-09-15T05:49:51.976 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1baea71e:::benchmark_data_smithi138_477915_object330:head# 2024-09-15T05:49:51.977 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1c0c4030:::benchmark_data_smithi138_477915_object325:head# 2024-09-15T05:49:51.977 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1ced30b9:::benchmark_data_smithi138_477915_object165:head# 2024-09-15T05:49:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1dac8c2e:::benchmark_data_smithi138_477915_object204:head# 2024-09-15T05:49:51.978 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1dc29afc:::benchmark_data_smithi138_477915_object398:head# 2024-09-15T05:49:51.979 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1f7be81b:::benchmark_data_smithi138_477915_object292:head# 2024-09-15T05:49:51.979 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1f84ed61:::benchmark_data_smithi138_483817_object172:head# 2024-09-15T05:49:51.980 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20000fdb:::benchmark_data_smithi138_477915_object157:head# 2024-09-15T05:49:51.980 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20909dac:::benchmark_data_smithi138_483817_object83:head# 2024-09-15T05:49:51.981 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20ce187a:::benchmark_data_smithi138_477915_object116:head# 2024-09-15T05:49:51.981 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:211ff0cd:::benchmark_data_smithi138_477915_object209:head# 2024-09-15T05:49:51.982 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:214fff42:::benchmark_data_smithi138_477915_object230:head# 2024-09-15T05:49:51.982 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:219197a2:::benchmark_data_smithi138_477915_object111:head# 2024-09-15T05:49:51.983 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:21be39a5:::benchmark_data_smithi138_483817_object31:head# 2024-09-15T05:49:51.983 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:22101c95:::benchmark_data_smithi138_477915_object59:head# 2024-09-15T05:49:51.984 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:223f5228:::benchmark_data_smithi138_477915_object79:head# 2024-09-15T05:49:51.984 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:22df1d28:::benchmark_data_smithi138_483817_object64:head# 2024-09-15T05:49:51.984 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:230de784:::benchmark_data_smithi138_477915_object244:head# 2024-09-15T05:49:51.985 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:246b4656:::benchmark_data_smithi138_477915_object191:head# 2024-09-15T05:49:51.985 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:25092a54:::benchmark_data_smithi138_477915_object10:head# 2024-09-15T05:49:51.985 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:251f2935:::benchmark_data_smithi138_483817_object49:head# 2024-09-15T05:49:51.986 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:263d2c7b:::benchmark_data_smithi138_477915_object153:head# 2024-09-15T05:49:51.986 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:266b4db5:::benchmark_data_smithi138_477915_object13:head# 2024-09-15T05:49:51.987 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26e68133:::benchmark_data_smithi138_477915_object349:head# 2024-09-15T05:49:51.988 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26f80018:::benchmark_data_smithi138_477915_object75:head# 2024-09-15T05:49:51.988 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2714c0b4:::benchmark_data_smithi138_483817_object156:head# 2024-09-15T05:49:51.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2729df23:::benchmark_data_smithi138_477915_object245:head# 2024-09-15T05:49:51.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:273a39c8:::benchmark_data_smithi138_477915_object212:head# 2024-09-15T05:49:51.989 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:287d6080:::benchmark_data_smithi138_483817_object70:head# 2024-09-15T05:49:51.990 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:29dae954:::benchmark_data_smithi138_483817_object117:head# 2024-09-15T05:49:51.990 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2a1a585d:::benchmark_data_smithi138_483817_object38:head# 2024-09-15T05:49:51.990 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2a9526df:::benchmark_data_smithi138_483817_object153:head# 2024-09-15T05:49:51.991 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2a962538:::benchmark_data_smithi138_483817_object101:head# 2024-09-15T05:49:51.991 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b0c02cc:::benchmark_data_smithi138_483817_object2:head# 2024-09-15T05:49:51.992 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b21cb84:::benchmark_data_smithi138_477915_object366:head# 2024-09-15T05:49:51.992 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b8191b4:::benchmark_data_smithi138_483817_object113:head# 2024-09-15T05:49:51.993 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b97e84e:::benchmark_data_smithi138_483817_object142:head# 2024-09-15T05:49:51.993 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2ba68ae5:::benchmark_data_smithi138_483817_object109:head# 2024-09-15T05:49:51.993 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2e0a6ba0:::benchmark_data_smithi138_477915_object45:head# 2024-09-15T05:49:51.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2ea804a3:::benchmark_data_smithi138_483817_object13:head# 2024-09-15T05:49:51.994 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2ecabff5:::benchmark_data_smithi138_483817_object106:head# 2024-09-15T05:49:51.995 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2eddd7ca:::benchmark_data_smithi138_483817_object129:head# 2024-09-15T05:49:51.995 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2f2263ed:::benchmark_data_smithi138_483817_object43:head# 2024-09-15T05:49:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2f2ab72c:::benchmark_data_smithi138_483817_object60:head# 2024-09-15T05:49:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2fd3771a:::benchmark_data_smithi138_477915_object326:head# 2024-09-15T05:49:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3097204c:::benchmark_data_smithi138_483817_object131:head# 2024-09-15T05:49:51.996 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:30b22779:::benchmark_data_smithi138_483817_object128:head# 2024-09-15T05:49:51.997 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:30de888f:::benchmark_data_smithi138_477915_object34:head# 2024-09-15T05:49:51.997 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:312dc4e7:::benchmark_data_smithi138_477915_object232:head# 2024-09-15T05:49:51.998 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:318ba285:::benchmark_data_smithi138_477915_object164:head# 2024-09-15T05:49:51.998 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:31e1d00e:::benchmark_data_smithi138_477915_object44:head# 2024-09-15T05:49:51.998 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:32a36681:::benchmark_data_smithi138_477915_object90:head# 2024-09-15T05:49:51.999 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3397aa63:::benchmark_data_smithi138_477915_object294:head# 2024-09-15T05:49:51.999 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:33bfd532:::benchmark_data_smithi138_483817_object176:head# 2024-09-15T05:49:52.000 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:350297ef:::benchmark_data_smithi138_477915_object382:head# 2024-09-15T05:49:52.001 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:35469d69:::benchmark_data_smithi138_483817_object132:head# 2024-09-15T05:49:52.001 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:354f77ae:::benchmark_data_smithi138_477915_object378:head# 2024-09-15T05:49:52.002 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:359485cb:::benchmark_data_smithi138_477915_object363:head# 2024-09-15T05:49:52.002 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:386459e6:::benchmark_data_smithi138_483817_object152:head# 2024-09-15T05:49:52.003 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:386dd901:::benchmark_data_smithi138_477915_object287:head# 2024-09-15T05:49:52.003 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:389afc62:::benchmark_data_smithi138_477915_object206:head# 2024-09-15T05:49:52.003 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38cb0823:::benchmark_data_smithi138_483817_object54:head# 2024-09-15T05:49:52.004 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38edce5f:::benchmark_data_smithi138_477915_object202:head# 2024-09-15T05:49:52.004 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:392023cd:::benchmark_data_smithi138_477915_object360:head# 2024-09-15T05:49:52.005 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3959ef6d:::benchmark_data_smithi138_483817_object66:head# 2024-09-15T05:49:52.005 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:396a3af9:::benchmark_data_smithi138_477915_object361:head# 2024-09-15T05:49:52.006 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:39870a18:::benchmark_data_smithi138_477915_object55:head# 2024-09-15T05:49:52.006 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:39d4ba45:::benchmark_data_smithi138_483817_object107:head# 2024-09-15T05:49:52.006 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3a34f987:::benchmark_data_smithi138_477915_object84:head# 2024-09-15T05:49:52.007 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3a45d3f0:::benchmark_data_smithi138_477915_object133:head# 2024-09-15T05:49:52.007 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3bc519ab:::benchmark_data_smithi138_477915_object301:head# 2024-09-15T05:49:52.007 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3bc57672:::benchmark_data_smithi138_477915_object317:head# 2024-09-15T05:49:52.008 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3d290d02:::benchmark_data_smithi138_477915_object52:head# 2024-09-15T05:49:52.009 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3d995ed5:::benchmark_data_smithi138_477915_object289:head# 2024-09-15T05:49:52.009 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3deada5a:::benchmark_data_smithi138_483817_object69:head# 2024-09-15T05:49:52.010 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3e073204:::benchmark_data_smithi138_477915_object47:head# 2024-09-15T05:49:52.010 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3eb24b0a:::benchmark_data_smithi138_483817_object127:head# 2024-09-15T05:49:52.011 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3f24c379:::benchmark_data_smithi138_477915_object381:head# 2024-09-15T05:49:52.011 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3f8ef9b0:::benchmark_data_smithi138_477915_object215:head# 2024-09-15T05:49:52.011 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4188c502:::benchmark_data_smithi138_477915_object394:head# 2024-09-15T05:49:52.012 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:41d0e70f:::benchmark_data_smithi138_477915_object124:head# 2024-09-15T05:49:52.012 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:42ce4ca2:::benchmark_data_smithi138_477915_object170:head# 2024-09-15T05:49:52.013 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4384dd5f:::benchmark_data_smithi138_477915_object222:head# 2024-09-15T05:49:52.013 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:43888adc:::benchmark_data_smithi138_483817_object137:head# 2024-09-15T05:49:52.014 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:441e81fc:::benchmark_data_smithi138_477915_object322:head# 2024-09-15T05:49:52.015 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:444f377d:::benchmark_data_smithi138_477915_object259:head# 2024-09-15T05:49:52.015 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44849f0f:::benchmark_data_smithi138_483817_object10:head# 2024-09-15T05:49:52.016 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:448c8ce2:::benchmark_data_smithi138_483817_object175:head# 2024-09-15T05:49:52.016 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44b71f94:::benchmark_data_smithi138_483817_object130:head# 2024-09-15T05:49:52.017 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44d1d1a6:::benchmark_data_smithi138_477915_object175:head# 2024-09-15T05:49:52.017 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44e95b23:::benchmark_data_smithi138_477915_object187:head# 2024-09-15T05:49:52.018 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44f216b2:::benchmark_data_smithi138_477915_object331:head# 2024-09-15T05:49:52.018 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:45612842:::benchmark_data_smithi138_483817_object6:head# 2024-09-15T05:49:52.019 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:45f59372:::benchmark_data_smithi138_477915_object134:head# 2024-09-15T05:49:52.019 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:463b99f0:::benchmark_data_smithi138_477915_object41:head# 2024-09-15T05:49:52.019 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4653fe80:::benchmark_data_smithi138_483817_object29:head# 2024-09-15T05:49:52.020 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:46e620bd:::benchmark_data_smithi138_483817_object42:head# 2024-09-15T05:49:52.020 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:46f42845:::benchmark_data_smithi138_477915_object50:head# 2024-09-15T05:49:52.021 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:47847711:::benchmark_data_smithi138_477915_object391:head# 2024-09-15T05:49:52.021 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:47e9ebca:::benchmark_data_smithi138_483817_object35:head# 2024-09-15T05:49:52.024 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:481a4dcf:::benchmark_data_smithi138_477915_object306:head# 2024-09-15T05:49:52.024 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48339ebf:::benchmark_data_smithi138_477915_object53:head# 2024-09-15T05:49:52.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48cd125a:::benchmark_data_smithi138_477915_object320:head# 2024-09-15T05:49:52.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:49297529:::benchmark_data_smithi138_477915_object110:head# 2024-09-15T05:49:52.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:492bb000:::benchmark_data_smithi138_477915_object364:head# 2024-09-15T05:49:52.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:49cc79d0:::benchmark_data_smithi138_483817_object67:head# 2024-09-15T05:49:52.025 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4a0a3214:::benchmark_data_smithi138_477915_object220:head# 2024-09-15T05:49:52.026 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4b9543c5:::benchmark_data_smithi138_477915_object1:head# 2024-09-15T05:49:52.026 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4ba0927c:::benchmark_data_smithi138_477915_object178:head# 2024-09-15T05:49:52.026 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4ba2f324:::benchmark_data_smithi138_477915_object293:head# 2024-09-15T05:49:52.026 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4c0b6957:::benchmark_data_smithi138_477915_object114:head# 2024-09-15T05:49:52.027 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4c11c9f7:::benchmark_data_smithi138_483817_object9:head# 2024-09-15T05:49:52.027 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4c7779cc:::benchmark_data_smithi138_477915_object379:head# 2024-09-15T05:49:52.028 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4cb16b08:::benchmark_data_smithi138_483817_object37:head# 2024-09-15T05:49:52.028 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4cffde3a:::benchmark_data_smithi138_477915_object271:head# 2024-09-15T05:49:52.029 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4d3925ed:::benchmark_data_smithi138_477915_object396:head# 2024-09-15T05:49:52.029 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4d67e2bb:::benchmark_data_smithi138_477915_object141:head# 2024-09-15T05:49:52.030 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4e93ce33:::benchmark_data_smithi138_477915_object82:head# 2024-09-15T05:49:52.030 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4eb39cc3:::benchmark_data_smithi138_477915_object11:head# 2024-09-15T05:49:52.030 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4f21a6e9:::benchmark_data_smithi138_483817_object98:head# 2024-09-15T05:49:52.031 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4fdb5ea1:::benchmark_data_smithi138_477915_object86:head# 2024-09-15T05:49:52.032 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4fea3b8e:::benchmark_data_smithi138_483817_object141:head# 2024-09-15T05:49:52.032 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52b28e51:::benchmark_data_smithi138_477915_object188:head# 2024-09-15T05:49:52.033 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52f1d98b:::benchmark_data_smithi138_477915_object29:head# 2024-09-15T05:49:52.033 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:535bd7b0:::benchmark_data_smithi138_483817_object21:head# 2024-09-15T05:49:52.033 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:538d02e4:::benchmark_data_smithi138_477915_object387:head# 2024-09-15T05:49:52.034 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:53b28f04:::benchmark_data_smithi138_477915_object193:head# 2024-09-15T05:49:52.034 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:543e73a1:::benchmark_data_smithi138_477915_object143:head# 2024-09-15T05:49:52.035 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:546413ea:::benchmark_data_smithi138_477915_object35:head# 2024-09-15T05:49:52.035 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54b5d110:::benchmark_data_smithi138_477915_object370:head# 2024-09-15T05:49:52.035 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54c3eb42:::benchmark_data_smithi138_483817_object171:head# 2024-09-15T05:49:52.036 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54d808a1:::benchmark_data_smithi138_477915_object147:head# 2024-09-15T05:49:52.036 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:556f515c:::benchmark_data_smithi138_477915_object367:head# 2024-09-15T05:49:52.037 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5581a3b9:::benchmark_data_smithi138_477915_object123:head# 2024-09-15T05:49:52.037 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:55b4ce8c:::benchmark_data_smithi138_477915_object327:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:55e0f391:::benchmark_data_smithi138_477915_object40:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:564dfa9a:::benchmark_data_smithi138_477915_object89:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56619472:::benchmark_data_smithi138_477915_object291:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56ac2e18:::benchmark_data_smithi138_477915_object373:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56e0ff0d:::benchmark_data_smithi138_477915_object246:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56e54ad7:::benchmark_data_smithi138_477915_object241:head# 2024-09-15T05:49:52.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:570d6bf9:::benchmark_data_smithi138_483817_object3:head# 2024-09-15T05:49:52.040 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:572a26cc:::benchmark_data_smithi138_477915_object3:head# 2024-09-15T05:49:52.040 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:57365c69:::benchmark_data_smithi138_477915_object185:head# 2024-09-15T05:49:52.040 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:58b05a98:::benchmark_data_smithi138_477915_object94:head# 2024-09-15T05:49:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5979f860:::benchmark_data_smithi138_477915_object42:head# 2024-09-15T05:49:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:598335b8:::benchmark_data_smithi138_483817_object114:head# 2024-09-15T05:49:52.041 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5a3930c3:::benchmark_data_smithi138_477915_object92:head# 2024-09-15T05:49:52.042 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5acdf762:::benchmark_data_smithi138_477915_object217:head# 2024-09-15T05:49:52.042 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5b192008:::benchmark_data_smithi138_477915_object190:head# 2024-09-15T05:49:52.043 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5bd647c9:::benchmark_data_smithi138_477915_object308:head# 2024-09-15T05:49:52.043 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5c5abf37:::benchmark_data_smithi138_477915_object227:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5cc75e94:::benchmark_data_smithi138_477915_object261:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5d452870:::benchmark_data_smithi138_477915_object386:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5e2e9211:::benchmark_data_smithi138_477915_object201:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5f10ef7e:::benchmark_data_smithi138_477915_object182:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fa8bc7a:::benchmark_data_smithi138_477915_object371:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fd3194b:::benchmark_data_smithi138_477915_object83:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5fec7540:::benchmark_data_smithi138_477915_object132:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6012bbb1:::benchmark_data_smithi138_477915_object213:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:620660f2:::benchmark_data_smithi138_477915_object138:head# 2024-09-15T05:49:52.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:62b5408f:::benchmark_data_smithi138_483817_object151:head# 2024-09-15T05:49:52.046 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:62fcea3a:::benchmark_data_smithi138_477915_object374:head# 2024-09-15T05:49:52.046 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:630b98f5:::benchmark_data_smithi138_483817_object80:head# 2024-09-15T05:49:52.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:635fcc02:::benchmark_data_smithi138_477915_object18:head# 2024-09-15T05:49:52.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:636ef2c0:::benchmark_data_smithi138_483817_object45:head# 2024-09-15T05:49:52.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:63be4c06:::benchmark_data_smithi138_483817_object1:head# 2024-09-15T05:49:52.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6429912b:::benchmark_data_smithi138_477915_object78:head# 2024-09-15T05:49:52.048 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:648cc3c2:::benchmark_data_smithi138_477915_object180:head# 2024-09-15T05:49:52.048 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:64ae8f93:::benchmark_data_smithi138_477915_object273:head# 2024-09-15T05:49:52.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:65851d9e:::benchmark_data_smithi138_477915_object276:head# 2024-09-15T05:49:52.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:66acc5ee:::benchmark_data_smithi138_477915_object155:head# 2024-09-15T05:49:52.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:677fba78:::benchmark_data_smithi138_483817_object91:head# 2024-09-15T05:49:52.050 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:67e15da6:::benchmark_data_smithi138_477915_object253:head# 2024-09-15T05:49:52.050 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68a5ecca:::benchmark_data_smithi138_477915_object260:head# 2024-09-15T05:49:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68b33428:::benchmark_data_smithi138_477915_object388:head# 2024-09-15T05:49:52.051 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6940b9d3:::benchmark_data_smithi138_477915_object239:head# 2024-09-15T05:49:52.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:69f7ef4c:::benchmark_data_smithi138_477915_object127:head# 2024-09-15T05:49:52.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6aef21a2:::benchmark_data_smithi138_477915_object54:head# 2024-09-15T05:49:52.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6b3c2b63:::benchmark_data_smithi138_477915_object208:head# 2024-09-15T05:49:52.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6bf4848f:::benchmark_data_smithi138_477915_object399:head# 2024-09-15T05:49:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6ccf34bf:::benchmark_data_smithi138_477915_object22:head# 2024-09-15T05:49:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6cec5432:::benchmark_data_smithi138_483817_object174:head# 2024-09-15T05:49:52.053 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6d5a3232:::benchmark_data_smithi138_477915_object43:head# 2024-09-15T05:49:52.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dc7f41b:::benchmark_data_smithi138_477915_object262:head# 2024-09-15T05:49:52.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dd17fb9:::benchmark_data_smithi138_477915_object263:head# 2024-09-15T05:49:52.055 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6ddcb643:::benchmark_data_smithi138_477915_object285:head# 2024-09-15T05:49:52.055 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dfa6e82:::benchmark_data_smithi138_483817_object44:head# 2024-09-15T05:49:52.055 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6e1716aa:::benchmark_data_smithi138_477915_object135:head# 2024-09-15T05:49:52.056 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:702ed571:::benchmark_data_smithi138_477915_object219:head# 2024-09-15T05:49:52.056 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:707bb828:::benchmark_data_smithi138_477915_object158:head# 2024-09-15T05:49:52.057 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:70ca4348:::benchmark_data_smithi138_477915_object26:head# 2024-09-15T05:49:52.057 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7100933a:::benchmark_data_smithi138_483817_object72:head# 2024-09-15T05:49:52.058 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:710e85f3:::benchmark_data_smithi138_483817_object32:head# 2024-09-15T05:49:52.058 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:713c8310:::benchmark_data_smithi138_483817_object71:head# 2024-09-15T05:49:52.058 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:71784a9a:::benchmark_data_smithi138_477915_object376:head# 2024-09-15T05:49:52.058 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:717dd36a:::benchmark_data_smithi138_477915_object139:head# 2024-09-15T05:49:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:71880930:::benchmark_data_smithi138_483817_object61:head# 2024-09-15T05:49:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:71fb5440:::benchmark_data_smithi138_477915_object97:head# 2024-09-15T05:49:52.060 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7314c8d4:::benchmark_data_smithi138_477915_object152:head# 2024-09-15T05:49:52.060 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:740159d0:::benchmark_data_smithi138_477915_object0:head# 2024-09-15T05:49:52.061 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:74886e53:::benchmark_data_smithi138_477915_object112:head# 2024-09-15T05:49:52.061 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:75046150:::benchmark_data_smithi138_477915_object279:head# 2024-09-15T05:49:52.062 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:758b6ae7:::benchmark_data_smithi138_483817_object148:head# 2024-09-15T05:49:52.063 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:76871f44:::benchmark_data_smithi138_483817_object5:head# 2024-09-15T05:49:52.063 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:773e28fe:::benchmark_data_smithi138_477915_object377:head# 2024-09-15T05:49:52.064 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7743c8d9:::benchmark_data_smithi138_483817_object24:head# 2024-09-15T05:49:52.064 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7783b60c:::benchmark_data_smithi138_477915_object101:head# 2024-09-15T05:49:52.064 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:786160e4:::benchmark_data_smithi138_477915_object38:head# 2024-09-15T05:49:52.065 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:78f9c0b4:::benchmark_data_smithi138_483817_object154:head# 2024-09-15T05:49:52.065 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7a2bef6b:::benchmark_data_smithi138_477915_object58:head# 2024-09-15T05:49:52.066 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7a76feec:::benchmark_data_smithi138_477915_object63:head# 2024-09-15T05:49:52.066 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7bb84b6a:::benchmark_data_smithi138_477915_object5:head# 2024-09-15T05:49:52.066 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7be49943:::benchmark_data_smithi138_477915_object48:head# 2024-09-15T05:49:52.067 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d1fdf77:::benchmark_data_smithi138_483817_object104:head# 2024-09-15T05:49:52.067 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d976229:::benchmark_data_smithi138_477915_object207:head# 2024-09-15T05:49:52.068 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d9d7a5f:::benchmark_data_smithi138_477915_object369:head# 2024-09-15T05:49:52.068 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7e2c4f9e:::benchmark_data_smithi138_483817_object173:head# 2024-09-15T05:49:52.069 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7eaec87b:::benchmark_data_smithi138_477915_object154:head# 2024-09-15T05:49:52.069 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7eda30d0:::benchmark_data_smithi138_477915_object310:head# 2024-09-15T05:49:52.070 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7f744066:::benchmark_data_smithi138_477915_object205:head# 2024-09-15T05:49:52.070 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81770d05:::benchmark_data_smithi138_483817_object103:head# 2024-09-15T05:49:52.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81cead58:::benchmark_data_smithi138_477915_object95:head# 2024-09-15T05:49:52.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81e31d0f:::benchmark_data_smithi138_477915_object203:head# 2024-09-15T05:49:52.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:829c9f42:::benchmark_data_smithi138_477915_object297:head# 2024-09-15T05:49:52.072 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8326356e:::benchmark_data_smithi138_477915_object272:head# 2024-09-15T05:49:52.073 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:84ba03a2:::benchmark_data_smithi138_477915_object268:head# 2024-09-15T05:49:52.073 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:84bb0071:::benchmark_data_smithi138_477915_object169:head# 2024-09-15T05:49:52.073 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8531dfd7:::benchmark_data_smithi138_477915_object311:head# 2024-09-15T05:49:52.074 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:853bf743:::benchmark_data_smithi138_477915_object257:head# 2024-09-15T05:49:52.074 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:861edb1c:::benchmark_data_smithi138_483817_object118:head# 2024-09-15T05:49:52.074 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:86e0b007:::benchmark_data_smithi138_477915_object74:head# 2024-09-15T05:49:52.075 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:86e93436:::benchmark_data_smithi138_477915_object319:head# 2024-09-15T05:49:52.075 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8709604b:::benchmark_data_smithi138_477915_object347:head# 2024-09-15T05:49:52.075 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:87d6852e:::benchmark_data_smithi138_477915_object231:head# 2024-09-15T05:49:52.075 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:88408030:::benchmark_data_smithi138_483817_object48:head# 2024-09-15T05:49:52.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:88c42840:::benchmark_data_smithi138_477915_object140:head# 2024-09-15T05:49:52.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8919b48c:::benchmark_data_smithi138_483817_object4:head# 2024-09-15T05:49:52.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8974e594:::benchmark_data_smithi138_483817_object115:head# 2024-09-15T05:49:52.077 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:89a81864:::benchmark_data_smithi138_477915_object264:head# 2024-09-15T05:49:52.077 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8a16f6c1:::benchmark_data_smithi138_477915_object93:head# 2024-09-15T05:49:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8a8bf090:::benchmark_data_smithi138_483817_object138:head# 2024-09-15T05:49:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8aa67a0a:::benchmark_data_smithi138_477915_object235:head# 2024-09-15T05:49:52.078 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8aedb92d:::benchmark_data_smithi138_477915_object365:head# 2024-09-15T05:49:52.079 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8b120672:::benchmark_data_smithi138_483817_object143:head# 2024-09-15T05:49:52.079 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8be95c11:::benchmark_data_smithi138_477915_object323:head# 2024-09-15T05:49:52.079 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8c3675a7:::benchmark_data_smithi138_477915_object303:head# 2024-09-15T05:49:52.080 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d8d0588:::benchmark_data_smithi138_477915_object28:head# 2024-09-15T05:49:52.080 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f046a5e:::benchmark_data_smithi138_477915_object109:head# 2024-09-15T05:49:52.080 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f04eb67:::benchmark_data_smithi138_477915_object160:head# 2024-09-15T05:49:52.080 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f226de4:::benchmark_data_smithi138_483817_object167:head# 2024-09-15T05:49:52.081 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8fca75c6:::benchmark_data_smithi138_483817_object160:head# 2024-09-15T05:49:52.081 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:900c1e9a:::benchmark_data_smithi138_477915_object168:head# 2024-09-15T05:49:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9033da55:::benchmark_data_smithi138_477915_object23:head# 2024-09-15T05:49:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90dfb114:::benchmark_data_smithi138_477915_object233:head# 2024-09-15T05:49:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90f41851:::benchmark_data_smithi138_477915_object256:head# 2024-09-15T05:49:52.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:916fb673:::benchmark_data_smithi138_483817_object88:head# 2024-09-15T05:49:52.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91c6592f:::benchmark_data_smithi138_477915_object36:head# 2024-09-15T05:49:52.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91f02d94:::benchmark_data_smithi138_477915_object249:head# 2024-09-15T05:49:52.084 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:92b0bd62:::benchmark_data_smithi138_483817_object73:head# 2024-09-15T05:49:52.084 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:92efb357:::benchmark_data_smithi138_477915_object242:head# 2024-09-15T05:49:52.085 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:937f5fff:::benchmark_data_smithi138_483817_object87:head# 2024-09-15T05:49:52.085 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9432f393:::benchmark_data_smithi138_477915_object16:head# 2024-09-15T05:49:52.085 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:945330ef:::benchmark_data_smithi138_483817_object51:head# 2024-09-15T05:49:52.085 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9459b9e3:::benchmark_data_smithi138_483817_object136:head# 2024-09-15T05:49:52.086 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:95f9f07e:::benchmark_data_smithi138_483817_object139:head# 2024-09-15T05:49:52.086 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:971a84a9:::benchmark_data_smithi138_477915_object149:head# 2024-09-15T05:49:52.087 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:971af863:::benchmark_data_smithi138_477915_object151:head# 2024-09-15T05:49:52.087 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:99020c09:::benchmark_data_smithi138_483817_object7:head# 2024-09-15T05:49:52.088 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a712198:::benchmark_data_smithi138_477915_object218:head# 2024-09-15T05:49:52.088 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9aa4d68e:::benchmark_data_smithi138_483817_object16:head# 2024-09-15T05:49:52.088 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9cba91af:::benchmark_data_smithi138_483817_object105:head# 2024-09-15T05:49:52.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9cbdb690:::benchmark_data_smithi138_483817_object20:head# 2024-09-15T05:49:52.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9d6d8ac9:::benchmark_data_smithi138_483817_object8:head# 2024-09-15T05:49:52.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9ddb669b:::benchmark_data_smithi138_483817_object65:head# 2024-09-15T05:49:52.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9df1f34f:::benchmark_data_smithi138_477915_object356:head# 2024-09-15T05:49:52.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9e3a6f9c:::benchmark_data_smithi138_483817_object40:head# 2024-09-15T05:49:52.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9e460c3c:::benchmark_data_smithi138_483817_object50:head# 2024-09-15T05:49:52.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9ea5731d:::benchmark_data_smithi138_483817_object79:head# 2024-09-15T05:49:52.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9eb64e24:::benchmark_data_smithi138_483817_object23:head# 2024-09-15T05:49:52.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f28ef18:::benchmark_data_smithi138_483817_object33:head# 2024-09-15T05:49:52.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f907d5d:::benchmark_data_smithi138_483817_object0:head# 2024-09-15T05:49:52.092 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f9adf98:::benchmark_data_smithi138_477915_object200:head# 2024-09-15T05:49:52.093 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a080aed2:::benchmark_data_smithi138_477915_object104:head# 2024-09-15T05:49:52.093 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0aeaa72:::benchmark_data_smithi138_477915_object254:head# 2024-09-15T05:49:52.093 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0d22f50:::benchmark_data_smithi138_477915_object71:head# 2024-09-15T05:49:52.093 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0fc5717:::benchmark_data_smithi138_483817_object53:head# 2024-09-15T05:49:52.093 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a10a53f5:::benchmark_data_smithi138_477915_object298:head# 2024-09-15T05:49:52.094 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a10dc166:::benchmark_data_smithi138_477915_object117:head# 2024-09-15T05:49:52.094 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a1460743:::benchmark_data_smithi138_477915_object334:head# 2024-09-15T05:49:52.094 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a14ca631:::benchmark_data_smithi138_477915_object105:head# 2024-09-15T05:49:52.095 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a1c26988:::benchmark_data_smithi138_477915_object351:head# 2024-09-15T05:49:52.095 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a2dc0f5b:::benchmark_data_smithi138_483817_object52:head# 2024-09-15T05:49:52.096 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a2e8a8e9:::benchmark_data_smithi138_477915_object184:head# 2024-09-15T05:49:52.096 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a300e90c:::benchmark_data_smithi138_483817_object155:head# 2024-09-15T05:49:52.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a5019f41:::benchmark_data_smithi138_477915_object302:head# 2024-09-15T05:49:52.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a5330bbb:::benchmark_data_smithi138_483817_object149:head# 2024-09-15T05:49:52.101 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a574d19f:::benchmark_data_smithi138_477915_object27:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a58e8ebc:::benchmark_data_smithi138_483817_object39:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a5add657:::benchmark_data_smithi138_477915_object96:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a65ba2e1:::benchmark_data_smithi138_477915_object348:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a76a8553:::benchmark_data_smithi138_477915_object384:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a84031a3:::benchmark_data_smithi138_477915_object156:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a89e1157:::benchmark_data_smithi138_477915_object66:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a8ec4680:::benchmark_data_smithi138_477915_object280:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a91194b3:::benchmark_data_smithi138_477915_object68:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a9405195:::benchmark_data_smithi138_483817_object25:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aad81857:::benchmark_data_smithi138_483817_object146:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aae671fe:::benchmark_data_smithi138_477915_object173:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:abb9f89a:::benchmark_data_smithi138_483817_object166:head# 2024-09-15T05:49:52.102 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:abf706c3:::benchmark_data_smithi138_477915_object121:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ad8a20c0:::benchmark_data_smithi138_477915_object375:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ade77a16:::benchmark_data_smithi138_477915_object88:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ae6263f9:::benchmark_data_smithi138_477915_object31:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aec18c78:::benchmark_data_smithi138_483817_object116:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:af390588:::benchmark_data_smithi138_483817_object41:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b02634ce:::benchmark_data_smithi138_483817_object108:head# 2024-09-15T05:49:52.104 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b05f7e87:::benchmark_data_smithi138_477915_object316:head# 2024-09-15T05:49:52.105 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b0627822:::benchmark_data_smithi138_477915_object255:head# 2024-09-15T05:49:52.105 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b08f0d28:::benchmark_data_smithi138_477915_object229:head# 2024-09-15T05:49:52.105 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b1344ffa:::benchmark_data_smithi138_477915_object33:head# 2024-09-15T05:49:52.106 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b142aea1:::benchmark_data_smithi138_477915_object64:head# 2024-09-15T05:49:52.106 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b2b44163:::benchmark_data_smithi138_483817_object74:head# 2024-09-15T05:49:52.106 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b2b53955:::benchmark_data_smithi138_477915_object199:head# 2024-09-15T05:49:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b2b94858:::benchmark_data_smithi138_477915_object345:head# 2024-09-15T05:49:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b3064d98:::benchmark_data_smithi138_483817_object26:head# 2024-09-15T05:49:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b30ed2d9:::benchmark_data_smithi138_477915_object224:head# 2024-09-15T05:49:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b36e2fb6:::benchmark_data_smithi138_477915_object56:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b388f84c:::benchmark_data_smithi138_477915_object270:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b3bb6a6a:::benchmark_data_smithi138_477915_object251:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b3ceb653:::benchmark_data_smithi138_477915_object130:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b413d550:::benchmark_data_smithi138_477915_object76:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b49d13a3:::benchmark_data_smithi138_477915_object296:head# 2024-09-15T05:49:52.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b4dcf7e3:::benchmark_data_smithi138_477915_object161:head# 2024-09-15T05:49:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b51b59a9:::benchmark_data_smithi138_477915_object159:head# 2024-09-15T05:49:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b52536fd:::benchmark_data_smithi138_477915_object102:head# 2024-09-15T05:49:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b711d581:::benchmark_data_smithi138_477915_object167:head# 2024-09-15T05:49:52.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b7914ecd:::benchmark_data_smithi138_477915_object343:head# 2024-09-15T05:49:52.111 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b7ce732e:::benchmark_data_smithi138_477915_object357:head# 2024-09-15T05:49:52.111 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8000b14:::benchmark_data_smithi138_477915_object103:head# 2024-09-15T05:49:52.112 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:49:52.112 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b89297e9:::benchmark_data_smithi138_477915_object362:head# 2024-09-15T05:49:52.112 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b895e95c:::benchmark_data_smithi138_477915_object359:head# 2024-09-15T05:49:52.112 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b897bd1c:::benchmark_data_smithi138_477915_object226:head# 2024-09-15T05:49:52.113 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8c52179:::benchmark_data_smithi138_483817_object125:head# 2024-09-15T05:49:52.113 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8f15cb8:::benchmark_data_smithi138_477915_object136:head# 2024-09-15T05:49:52.114 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bacecf10:::benchmark_data_smithi138_477915_object340:head# 2024-09-15T05:49:52.114 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bcda79f2:::benchmark_data_smithi138_477915_object228:head# 2024-09-15T05:49:52.114 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bdbfac57:::benchmark_data_smithi138_477915_object171:head# 2024-09-15T05:49:52.114 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bde3dd73:::benchmark_data_smithi138_483817_object145:head# 2024-09-15T05:49:52.114 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:be946f7d:::benchmark_data_smithi138_477915_object390:head# 2024-09-15T05:49:52.115 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bf91c211:::benchmark_data_smithi138_477915_object162:head# 2024-09-15T05:49:52.120 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bf9471cf:::benchmark_data_smithi138_483817_object144:head# 2024-09-15T05:49:52.120 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c080fdda:::benchmark_data_smithi138_477915_object354:head# 2024-09-15T05:49:52.121 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c0be2264:::benchmark_data_smithi138_483817_object120:head# 2024-09-15T05:49:52.121 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c0cda59f:::benchmark_data_smithi138_483817_object57:head# 2024-09-15T05:49:52.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c11f897f:::benchmark_data_smithi138_477915_object236:head# 2024-09-15T05:49:52.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c122f290:::benchmark_data_smithi138_477915_object172:head# 2024-09-15T05:49:52.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c1ed322f:::benchmark_data_smithi138_477915_object129:head# 2024-09-15T05:49:52.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c2727a0a:::benchmark_data_smithi138_483817_object163:head# 2024-09-15T05:49:52.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c2bef58a:::benchmark_data_smithi138_477915_object238:head# 2024-09-15T05:49:52.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c2e2e8b4:::benchmark_data_smithi138_477915_object234:head# 2024-09-15T05:49:52.125 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c36b3bbe:::benchmark_data_smithi138_477915_object247:head# 2024-09-15T05:49:52.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c38e75c6:::benchmark_data_smithi138_483817_object119:head# 2024-09-15T05:49:52.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c46189c0:::benchmark_data_smithi138_477915_object300:head# 2024-09-15T05:49:52.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c4b7580a:::benchmark_data_smithi138_483817_object100:head# 2024-09-15T05:49:52.127 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c4c0edd9:::benchmark_data_smithi138_477915_object87:head# 2024-09-15T05:49:52.128 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c4e6b38c:::benchmark_data_smithi138_483817_object110:head# 2024-09-15T05:49:52.128 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5229070:::benchmark_data_smithi138_477915_object237:head# 2024-09-15T05:49:52.128 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5a6edda:::benchmark_data_smithi138_483817_object112:head# 2024-09-15T05:49:52.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5aafc40:::benchmark_data_smithi138_477915_object21:head# 2024-09-15T05:49:52.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5bcceac:::benchmark_data_smithi138_477915_object314:head# 2024-09-15T05:49:52.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5cf901b:::benchmark_data_smithi138_477915_object69:head# 2024-09-15T05:49:52.130 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c612e5a0:::benchmark_data_smithi138_477915_object131:head# 2024-09-15T05:49:52.130 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c686c228:::benchmark_data_smithi138_477915_object383:head# 2024-09-15T05:49:52.131 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c6d1efb1:::benchmark_data_smithi138_477915_object269:head# 2024-09-15T05:49:52.131 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7b09ada:::benchmark_data_smithi138_477915_object350:head# 2024-09-15T05:49:52.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7e9582c:::benchmark_data_smithi138_477915_object318:head# 2024-09-15T05:49:52.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8129485:::benchmark_data_smithi138_477915_object400:head# 2024-09-15T05:49:52.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c81375d4:::benchmark_data_smithi138_477915_object186:head# 2024-09-15T05:49:52.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c81e2b79:::benchmark_data_smithi138_483817_object95:head# 2024-09-15T05:49:52.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c836a7f2:::benchmark_data_smithi138_477915_object392:head# 2024-09-15T05:49:52.133 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c84b5188:::benchmark_data_smithi138_477915_object282:head# 2024-09-15T05:49:52.133 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8ed4655:::benchmark_data_smithi138_483817_object28:head# 2024-09-15T05:49:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8f99db7:::benchmark_data_smithi138_477915_object295:head# 2024-09-15T05:49:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c918d639:::benchmark_data_smithi138_483817_object55:head# 2024-09-15T05:49:52.134 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c94caa86:::benchmark_data_smithi138_483817_object140:head# 2024-09-15T05:49:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c957d3d7:::benchmark_data_smithi138_483817_object59:head# 2024-09-15T05:49:52.135 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ca8c3be3:::benchmark_data_smithi138_483817_object134:head# 2024-09-15T05:49:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cac2fc5a:::benchmark_data_smithi138_477915_object380:head# 2024-09-15T05:49:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cb485fde:::benchmark_data_smithi138_477915_object342:head# 2024-09-15T05:49:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cb97c1ba:::benchmark_data_smithi138_483817_object165:head# 2024-09-15T05:49:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cbb3fcfe:::benchmark_data_smithi138_477915_object258:head# 2024-09-15T05:49:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd46d84a:::benchmark_data_smithi138_477915_object337:head# 2024-09-15T05:49:52.138 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd70a88a:::benchmark_data_smithi138_483817_object161:head# 2024-09-15T05:49:52.138 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd852d6c:::benchmark_data_smithi138_477915_object107:head# 2024-09-15T05:49:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd8a05cc:::benchmark_data_smithi138_477915_object2:head# 2024-09-15T05:49:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ce5af95b:::benchmark_data_smithi138_483817_object147:head# 2024-09-15T05:49:52.139 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ce8b7145:::benchmark_data_smithi138_483817_object78:head# 2024-09-15T05:49:52.140 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cec5b2bb:::benchmark_data_smithi138_477915_object25:head# 2024-09-15T05:49:52.140 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d0e38665:::benchmark_data_smithi138_477915_object266:head# 2024-09-15T05:49:52.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d165ae0f:::benchmark_data_smithi138_477915_object211:head# 2024-09-15T05:49:52.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d20fbe61:::benchmark_data_smithi138_477915_object284:head# 2024-09-15T05:49:52.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d2de79dc:::benchmark_data_smithi138_483817_object94:head# 2024-09-15T05:49:52.142 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d32e8031:::benchmark_data_smithi138_483817_object81:head# 2024-09-15T05:49:52.142 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d350cfaa:::benchmark_data_smithi138_477915_object113:head# 2024-09-15T05:49:52.142 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d3967d82:::benchmark_data_smithi138_483817_object58:head# 2024-09-15T05:49:52.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d3b5e940:::benchmark_data_smithi138_477915_object14:head# 2024-09-15T05:49:52.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d3b64364:::benchmark_data_smithi138_483817_object36:head# 2024-09-15T05:49:52.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d41bfbfa:::benchmark_data_smithi138_483817_object12:head# 2024-09-15T05:49:52.144 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d48aafda:::benchmark_data_smithi138_477915_object358:head# 2024-09-15T05:49:52.144 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d567f524:::benchmark_data_smithi138_483817_object169:head# 2024-09-15T05:49:52.144 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d5a212a5:::benchmark_data_smithi138_477915_object106:head# 2024-09-15T05:49:52.145 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d5d6c19c:::benchmark_data_smithi138_477915_object355:head# 2024-09-15T05:49:52.146 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d6691603:::benchmark_data_smithi138_483817_object22:head# 2024-09-15T05:49:52.146 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d6d89b5a:::benchmark_data_smithi138_477915_object341:head# 2024-09-15T05:49:52.146 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d789c5f7:::benchmark_data_smithi138_477915_object336:head# 2024-09-15T05:49:52.147 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7c4ab31:::benchmark_data_smithi138_477915_object189:head# 2024-09-15T05:49:52.147 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7c508f3:::benchmark_data_smithi138_477915_object4:head# 2024-09-15T05:49:52.147 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7ebcc2e:::benchmark_data_smithi138_477915_object265:head# 2024-09-15T05:49:52.148 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d878fde1:::benchmark_data_smithi138_483817_object93:head# 2024-09-15T05:49:52.148 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8daf539:::benchmark_data_smithi138_477915_object99:head# 2024-09-15T05:49:52.149 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8e85bad:::benchmark_data_smithi138_477915_object166:head# 2024-09-15T05:49:52.149 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d95dedd9:::benchmark_data_smithi138_477915_object274:head# 2024-09-15T05:49:52.149 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da010958:::benchmark_data_smithi138_483817_object85:head# 2024-09-15T05:49:52.150 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da2f71a7:::benchmark_data_smithi138_477915_object65:head# 2024-09-15T05:49:52.150 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da407726:::benchmark_data_smithi138_477915_object179:head# 2024-09-15T05:49:52.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da7cec77:::benchmark_data_smithi138_483817_object102:head# 2024-09-15T05:49:52.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dabfd9f7:::benchmark_data_smithi138_477915_object118:head# 2024-09-15T05:49:52.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dad5a76a:::benchmark_data_smithi138_477915_object305:head# 2024-09-15T05:49:52.152 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd1177ff:::benchmark_data_smithi138_483817_object47:head# 2024-09-15T05:49:52.152 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd6aea31:::benchmark_data_smithi138_477915_object150:head# 2024-09-15T05:49:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd78a891:::benchmark_data_smithi138_477915_object353:head# 2024-09-15T05:49:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:df8c54da:::benchmark_data_smithi138_483817_object111:head# 2024-09-15T05:49:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dfa1d077:::benchmark_data_smithi138_483817_object17:head# 2024-09-15T05:49:52.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dfac43cc:::benchmark_data_smithi138_477915_object126:head# 2024-09-15T05:49:52.154 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dfe8aa68:::benchmark_data_smithi138_477915_object51:head# 2024-09-15T05:49:52.154 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e0895a86:::benchmark_data_smithi138_483817_object158:head# 2024-09-15T05:49:52.155 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e100c584:::benchmark_data_smithi138_477915_object8:head# 2024-09-15T05:49:52.155 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e1828df5:::benchmark_data_smithi138_477915_object267:head# 2024-09-15T05:49:52.156 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e1ad5afa:::benchmark_data_smithi138_483817_object30:head# 2024-09-15T05:49:52.156 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e1daa4ae:::benchmark_data_smithi138_477915_object37:head# 2024-09-15T05:49:52.156 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e23cd4da:::benchmark_data_smithi138_477915_object32:head# 2024-09-15T05:49:52.157 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e436bdb7:::benchmark_data_smithi138_483817_object68:head# 2024-09-15T05:49:52.157 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e4da1a27:::benchmark_data_smithi138_477915_object15:head# 2024-09-15T05:49:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e4f1a68f:::benchmark_data_smithi138_477915_object395:head# 2024-09-15T05:49:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e575c0b5:::benchmark_data_smithi138_483817_object157:head# 2024-09-15T05:49:52.158 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e6fdeca3:::benchmark_data_smithi138_477915_object338:head# 2024-09-15T05:49:52.159 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e7c862ab:::benchmark_data_smithi138_477915_object119:head# 2024-09-15T05:49:52.159 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e7e5da90:::benchmark_data_smithi138_477915_object393:head# 2024-09-15T05:49:52.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e922bb7a:::benchmark_data_smithi138_483817_object162:head# 2024-09-15T05:49:52.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e9e1dfd6:::benchmark_data_smithi138_477915_object315:head# 2024-09-15T05:49:52.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eaee7ade:::benchmark_data_smithi138_477915_object277:head# 2024-09-15T05:49:52.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eb2d0c81:::benchmark_data_smithi138_477915_object352:head# 2024-09-15T05:49:52.161 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eb74e80f:::benchmark_data_smithi138_477915_object39:head# 2024-09-15T05:49:52.161 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ebaa8086:::benchmark_data_smithi138_483817_object123:head# 2024-09-15T05:49:52.162 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ebd302e1:::benchmark_data_smithi138_477915_object221:head# 2024-09-15T05:49:52.162 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ebde0c8a:::benchmark_data_smithi138_483817_object170:head# 2024-09-15T05:49:52.162 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec58d5d5:::benchmark_data_smithi138_483817_object97:head# 2024-09-15T05:49:52.163 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec96583a:::benchmark_data_smithi138_477915_object6:head# 2024-09-15T05:49:52.164 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ecd13f7c:::benchmark_data_smithi138_477915_object174:head# 2024-09-15T05:49:52.164 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ecda3af5:::benchmark_data_smithi138_477915_object7:head# 2024-09-15T05:49:52.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed2124e8:::benchmark_data_smithi138_477915_object283:head# 2024-09-15T05:49:52.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed4f8904:::benchmark_data_smithi138_477915_object321:head# 2024-09-15T05:49:52.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eec767c6:::benchmark_data_smithi138_477915_object304:head# 2024-09-15T05:49:52.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eedebaa9:::benchmark_data_smithi138_483817_object121:head# 2024-09-15T05:49:52.166 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef20ff5b:::benchmark_data_smithi138_477915_object148:head# 2024-09-15T05:49:52.166 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef304116:::benchmark_data_smithi138_477915_object49:head# 2024-09-15T05:49:52.167 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef4d9672:::benchmark_data_smithi138_483817_object126:head# 2024-09-15T05:49:52.167 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef573daf:::benchmark_data_smithi138_477915_object177:head# 2024-09-15T05:49:52.167 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:effa6446:::benchmark_data_smithi138_483817_object150:head# 2024-09-15T05:49:52.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f0690f60:::benchmark_data_smithi138_477915_object397:head# 2024-09-15T05:49:52.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f15d302a:::benchmark_data_smithi138_477915_object81:head# 2024-09-15T05:49:52.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f2a7b9b6:::benchmark_data_smithi138_483817_object164:head# 2024-09-15T05:49:52.169 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f2b2bc3f:::benchmark_data_smithi138_477915_object216:head# 2024-09-15T05:49:52.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f314e879:::benchmark_data_smithi138_477915_object290:head# 2024-09-15T05:49:52.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f358deca:::benchmark_data_smithi138_477915_object278:head# 2024-09-15T05:49:52.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f35a9e64:::benchmark_data_smithi138_483817_object84:head# 2024-09-15T05:49:52.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f3ae3ae7:::benchmark_data_smithi138_483817_object14:head# 2024-09-15T05:49:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f40ce7f6:::benchmark_data_smithi138_477915_object9:head# 2024-09-15T05:49:52.171 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f48dff42:::benchmark_data_smithi138_477915_object372:head# 2024-09-15T05:49:52.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4a59f6a:::benchmark_data_smithi138_477915_object12:head# 2024-09-15T05:49:52.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4d52c7d:::benchmark_data_smithi138_477915_object183:head# 2024-09-15T05:49:52.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4ffa0f0:::benchmark_data_smithi138_483817_object27:head# 2024-09-15T05:49:52.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f50e2ea1:::benchmark_data_smithi138_477915_object194:head# 2024-09-15T05:49:52.173 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5165677:::benchmark_data_smithi138_477915_object346:head# 2024-09-15T05:49:52.173 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5ba49b3:::benchmark_data_smithi138_477915_object333:head# 2024-09-15T05:49:52.174 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5c42dd1:::benchmark_data_smithi138_477915_object85:head# 2024-09-15T05:49:52.174 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f64f6100:::benchmark_data_smithi138_477915_object163:head# 2024-09-15T05:49:52.174 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f664855f:::benchmark_data_smithi138_483817_object90:head# 2024-09-15T05:49:52.176 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f6daa659:::benchmark_data_smithi138_477915_object250:head# 2024-09-15T05:49:52.176 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f7f49d03:::benchmark_data_smithi138_477915_object324:head# 2024-09-15T05:49:52.176 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f956b667:::benchmark_data_smithi138_483817_object89:head# 2024-09-15T05:49:52.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f9630340:::benchmark_data_smithi138_483817_object46:head# 2024-09-15T05:49:52.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f964d056:::benchmark_data_smithi138_483817_object159:head# 2024-09-15T05:49:52.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa26fae3:::benchmark_data_smithi138_477915_object73:head# 2024-09-15T05:49:52.178 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa6cc2ae:::benchmark_data_smithi138_477915_object91:head# 2024-09-15T05:49:52.178 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa9ab79f:::benchmark_data_smithi138_477915_object72:head# 2024-09-15T05:49:52.179 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fb6a8c6a:::benchmark_data_smithi138_477915_object122:head# 2024-09-15T05:49:52.179 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fba968a6:::benchmark_data_smithi138_477915_object368:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fd402ec6:::benchmark_data_smithi138_477915_object67:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fdef6d22:::benchmark_data_smithi138_477915_object309:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fe273b33:::benchmark_data_smithi138_477915_object146:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fea7c25b:::benchmark_data_smithi138_477915_object344:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ff1959bb:::benchmark_data_smithi138_477915_object80:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffb3fc0b:::benchmark_data_smithi138_477915_object115:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffbe43c0:::benchmark_data_smithi138_477915_object125:head# 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T05:49:52.181 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:49:52.182 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:49:52.182 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T05:49:53.083 INFO:tasks.workunit.client.0.smithi138.stderr:Remove successful 2024-09-15T05:49:53.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-09-15T05:49:54.855 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:49:54.855 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.1 2024-09-15T05:49:54.855 INFO:tasks.workunit.client.0.smithi138.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-15T05:49:54.855 INFO:tasks.workunit.client.0.smithi138.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-15T05:49:55.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-15T05:49:56.547 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:49:56.547 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:49:56.547 INFO:tasks.workunit.client.0.smithi138.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-15T05:49:56.547 INFO:tasks.workunit.client.0.smithi138.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-09-15T05:49:57.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-09-15T05:49:58.219 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:49:58.219 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.1 2024-09-15T05:49:58.219 INFO:tasks.workunit.client.0.smithi138.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-15T05:49:58.220 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 2 -> 1 2024-09-15T05:49:58.220 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81cead58:::benchmark_data_smithi138_477915_object95:head# 2024-09-15T05:49:58.220 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.275 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81e31d0f:::benchmark_data_smithi138_477915_object203:head# 2024-09-15T05:49:58.275 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.325 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:829c9f42:::benchmark_data_smithi138_477915_object297:head# 2024-09-15T05:49:58.325 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.359 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8326356e:::benchmark_data_smithi138_477915_object272:head# 2024-09-15T05:49:58.359 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.392 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84ba03a2:::benchmark_data_smithi138_477915_object268:head# 2024-09-15T05:49:58.393 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.426 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84bb0071:::benchmark_data_smithi138_477915_object169:head# 2024-09-15T05:49:58.426 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.460 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8531dfd7:::benchmark_data_smithi138_477915_object311:head# 2024-09-15T05:49:58.460 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.493 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:853bf743:::benchmark_data_smithi138_477915_object257:head# 2024-09-15T05:49:58.493 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.543 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e0b007:::benchmark_data_smithi138_477915_object74:head# 2024-09-15T05:49:58.544 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.594 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e93436:::benchmark_data_smithi138_477915_object319:head# 2024-09-15T05:49:58.594 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.644 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8709604b:::benchmark_data_smithi138_477915_object347:head# 2024-09-15T05:49:58.644 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.728 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:87d6852e:::benchmark_data_smithi138_477915_object231:head# 2024-09-15T05:49:58.728 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.795 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:88c42840:::benchmark_data_smithi138_477915_object140:head# 2024-09-15T05:49:58.795 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.845 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:89a81864:::benchmark_data_smithi138_477915_object264:head# 2024-09-15T05:49:58.845 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.895 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a16f6c1:::benchmark_data_smithi138_477915_object93:head# 2024-09-15T05:49:58.896 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.945 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aa67a0a:::benchmark_data_smithi138_477915_object235:head# 2024-09-15T05:49:58.946 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:58.996 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aedb92d:::benchmark_data_smithi138_477915_object365:head# 2024-09-15T05:49:58.996 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.046 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8be95c11:::benchmark_data_smithi138_477915_object323:head# 2024-09-15T05:49:59.046 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.096 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c3675a7:::benchmark_data_smithi138_477915_object303:head# 2024-09-15T05:49:59.096 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.155 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d8d0588:::benchmark_data_smithi138_477915_object28:head# 2024-09-15T05:49:59.155 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.247 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f046a5e:::benchmark_data_smithi138_477915_object109:head# 2024-09-15T05:49:59.247 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.339 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f04eb67:::benchmark_data_smithi138_477915_object160:head# 2024-09-15T05:49:59.339 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.406 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:900c1e9a:::benchmark_data_smithi138_477915_object168:head# 2024-09-15T05:49:59.407 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.457 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9033da55:::benchmark_data_smithi138_477915_object23:head# 2024-09-15T05:49:59.457 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90dfb114:::benchmark_data_smithi138_477915_object233:head# 2024-09-15T05:49:59.507 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.557 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90f41851:::benchmark_data_smithi138_477915_object256:head# 2024-09-15T05:49:59.558 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.607 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91c6592f:::benchmark_data_smithi138_477915_object36:head# 2024-09-15T05:49:59.607 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.666 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91f02d94:::benchmark_data_smithi138_477915_object249:head# 2024-09-15T05:49:59.666 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.716 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92efb357:::benchmark_data_smithi138_477915_object242:head# 2024-09-15T05:49:59.717 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.766 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9432f393:::benchmark_data_smithi138_477915_object16:head# 2024-09-15T05:49:59.767 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.817 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971a84a9:::benchmark_data_smithi138_477915_object149:head# 2024-09-15T05:49:59.817 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.867 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971af863:::benchmark_data_smithi138_477915_object151:head# 2024-09-15T05:49:59.867 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.909 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a712198:::benchmark_data_smithi138_477915_object218:head# 2024-09-15T05:49:59.909 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.951 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9df1f34f:::benchmark_data_smithi138_477915_object356:head# 2024-09-15T05:49:59.951 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:49:59.993 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f9adf98:::benchmark_data_smithi138_477915_object200:head# 2024-09-15T05:49:59.993 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.043 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a080aed2:::benchmark_data_smithi138_477915_object104:head# 2024-09-15T05:50:00.043 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.076 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0aeaa72:::benchmark_data_smithi138_477915_object254:head# 2024-09-15T05:50:00.077 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.110 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0d22f50:::benchmark_data_smithi138_477915_object71:head# 2024-09-15T05:50:00.110 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.144 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10a53f5:::benchmark_data_smithi138_477915_object298:head# 2024-09-15T05:50:00.144 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.177 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10dc166:::benchmark_data_smithi138_477915_object117:head# 2024-09-15T05:50:00.177 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.211 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1460743:::benchmark_data_smithi138_477915_object334:head# 2024-09-15T05:50:00.211 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.261 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a14ca631:::benchmark_data_smithi138_477915_object105:head# 2024-09-15T05:50:00.261 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.311 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1c26988:::benchmark_data_smithi138_477915_object351:head# 2024-09-15T05:50:00.312 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.361 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a2e8a8e9:::benchmark_data_smithi138_477915_object184:head# 2024-09-15T05:50:00.362 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.412 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5019f41:::benchmark_data_smithi138_477915_object302:head# 2024-09-15T05:50:00.412 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.462 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a574d19f:::benchmark_data_smithi138_477915_object27:head# 2024-09-15T05:50:00.462 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.512 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5add657:::benchmark_data_smithi138_477915_object96:head# 2024-09-15T05:50:00.512 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.562 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a65ba2e1:::benchmark_data_smithi138_477915_object348:head# 2024-09-15T05:50:00.562 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.613 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a76a8553:::benchmark_data_smithi138_477915_object384:head# 2024-09-15T05:50:00.613 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.663 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a84031a3:::benchmark_data_smithi138_477915_object156:head# 2024-09-15T05:50:00.663 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.764 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a89e1157:::benchmark_data_smithi138_477915_object66:head# 2024-09-15T05:50:00.764 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a8ec4680:::benchmark_data_smithi138_477915_object280:head# 2024-09-15T05:50:00.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.864 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a91194b3:::benchmark_data_smithi138_477915_object68:head# 2024-09-15T05:50:00.865 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:00.931 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aae671fe:::benchmark_data_smithi138_477915_object173:head# 2024-09-15T05:50:00.931 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.006 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:abf706c3:::benchmark_data_smithi138_477915_object121:head# 2024-09-15T05:50:01.007 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.057 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ad8a20c0:::benchmark_data_smithi138_477915_object375:head# 2024-09-15T05:50:01.057 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.107 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ade77a16:::benchmark_data_smithi138_477915_object88:head# 2024-09-15T05:50:01.107 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.157 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ae6263f9:::benchmark_data_smithi138_477915_object31:head# 2024-09-15T05:50:01.157 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.249 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b05f7e87:::benchmark_data_smithi138_477915_object316:head# 2024-09-15T05:50:01.250 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.341 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b0627822:::benchmark_data_smithi138_477915_object255:head# 2024-09-15T05:50:01.342 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.409 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b08f0d28:::benchmark_data_smithi138_477915_object229:head# 2024-09-15T05:50:01.409 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b1344ffa:::benchmark_data_smithi138_477915_object33:head# 2024-09-15T05:50:01.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.509 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b142aea1:::benchmark_data_smithi138_477915_object64:head# 2024-09-15T05:50:01.509 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.559 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b53955:::benchmark_data_smithi138_477915_object199:head# 2024-09-15T05:50:01.559 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.601 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b94858:::benchmark_data_smithi138_477915_object345:head# 2024-09-15T05:50:01.601 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.643 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b30ed2d9:::benchmark_data_smithi138_477915_object224:head# 2024-09-15T05:50:01.643 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.693 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b36e2fb6:::benchmark_data_smithi138_477915_object56:head# 2024-09-15T05:50:01.694 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.743 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b388f84c:::benchmark_data_smithi138_477915_object270:head# 2024-09-15T05:50:01.744 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.777 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3bb6a6a:::benchmark_data_smithi138_477915_object251:head# 2024-09-15T05:50:01.777 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.811 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3ceb653:::benchmark_data_smithi138_477915_object130:head# 2024-09-15T05:50:01.811 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.844 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b413d550:::benchmark_data_smithi138_477915_object76:head# 2024-09-15T05:50:01.844 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.886 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b49d13a3:::benchmark_data_smithi138_477915_object296:head# 2024-09-15T05:50:01.886 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.920 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b4dcf7e3:::benchmark_data_smithi138_477915_object161:head# 2024-09-15T05:50:01.920 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:01.970 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b51b59a9:::benchmark_data_smithi138_477915_object159:head# 2024-09-15T05:50:01.970 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.020 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b52536fd:::benchmark_data_smithi138_477915_object102:head# 2024-09-15T05:50:02.020 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.112 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b711d581:::benchmark_data_smithi138_477915_object167:head# 2024-09-15T05:50:02.113 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.204 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7914ecd:::benchmark_data_smithi138_477915_object343:head# 2024-09-15T05:50:02.205 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.322 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7ce732e:::benchmark_data_smithi138_477915_object357:head# 2024-09-15T05:50:02.323 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8000b14:::benchmark_data_smithi138_477915_object103:head# 2024-09-15T05:50:02.423 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.515 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:50:02.515 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.640 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b89297e9:::benchmark_data_smithi138_477915_object362:head# 2024-09-15T05:50:02.640 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.749 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b895e95c:::benchmark_data_smithi138_477915_object359:head# 2024-09-15T05:50:02.749 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.824 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b897bd1c:::benchmark_data_smithi138_477915_object226:head# 2024-09-15T05:50:02.825 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.874 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8f15cb8:::benchmark_data_smithi138_477915_object136:head# 2024-09-15T05:50:02.875 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.925 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bacecf10:::benchmark_data_smithi138_477915_object340:head# 2024-09-15T05:50:02.925 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:02.975 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bcda79f2:::benchmark_data_smithi138_477915_object228:head# 2024-09-15T05:50:02.975 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.025 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bdbfac57:::benchmark_data_smithi138_477915_object171:head# 2024-09-15T05:50:03.026 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.075 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:be946f7d:::benchmark_data_smithi138_477915_object390:head# 2024-09-15T05:50:03.076 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.126 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bf91c211:::benchmark_data_smithi138_477915_object162:head# 2024-09-15T05:50:03.126 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.176 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c080fdda:::benchmark_data_smithi138_477915_object354:head# 2024-09-15T05:50:03.176 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.268 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c11f897f:::benchmark_data_smithi138_477915_object236:head# 2024-09-15T05:50:03.268 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.344 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c122f290:::benchmark_data_smithi138_477915_object172:head# 2024-09-15T05:50:03.344 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.377 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c1ed322f:::benchmark_data_smithi138_477915_object129:head# 2024-09-15T05:50:03.378 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2bef58a:::benchmark_data_smithi138_477915_object238:head# 2024-09-15T05:50:03.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.444 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2e2e8b4:::benchmark_data_smithi138_477915_object234:head# 2024-09-15T05:50:03.445 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.478 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c36b3bbe:::benchmark_data_smithi138_477915_object247:head# 2024-09-15T05:50:03.478 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.520 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c46189c0:::benchmark_data_smithi138_477915_object300:head# 2024-09-15T05:50:03.520 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.570 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4c0edd9:::benchmark_data_smithi138_477915_object87:head# 2024-09-15T05:50:03.570 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.620 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5229070:::benchmark_data_smithi138_477915_object237:head# 2024-09-15T05:50:03.621 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.671 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5aafc40:::benchmark_data_smithi138_477915_object21:head# 2024-09-15T05:50:03.671 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.721 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5bcceac:::benchmark_data_smithi138_477915_object314:head# 2024-09-15T05:50:03.721 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.771 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5cf901b:::benchmark_data_smithi138_477915_object69:head# 2024-09-15T05:50:03.771 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.830 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c612e5a0:::benchmark_data_smithi138_477915_object131:head# 2024-09-15T05:50:03.830 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.880 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c686c228:::benchmark_data_smithi138_477915_object383:head# 2024-09-15T05:50:03.881 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.931 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c6d1efb1:::benchmark_data_smithi138_477915_object269:head# 2024-09-15T05:50:03.931 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:03.981 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7b09ada:::benchmark_data_smithi138_477915_object350:head# 2024-09-15T05:50:03.981 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.031 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7e9582c:::benchmark_data_smithi138_477915_object318:head# 2024-09-15T05:50:04.031 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.081 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8129485:::benchmark_data_smithi138_477915_object400:head# 2024-09-15T05:50:04.082 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.132 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c81375d4:::benchmark_data_smithi138_477915_object186:head# 2024-09-15T05:50:04.132 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.182 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c836a7f2:::benchmark_data_smithi138_477915_object392:head# 2024-09-15T05:50:04.182 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.232 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c84b5188:::benchmark_data_smithi138_477915_object282:head# 2024-09-15T05:50:04.232 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.291 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8f99db7:::benchmark_data_smithi138_477915_object295:head# 2024-09-15T05:50:04.291 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.341 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cac2fc5a:::benchmark_data_smithi138_477915_object380:head# 2024-09-15T05:50:04.341 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.391 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cb485fde:::benchmark_data_smithi138_477915_object342:head# 2024-09-15T05:50:04.391 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.441 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cbb3fcfe:::benchmark_data_smithi138_477915_object258:head# 2024-09-15T05:50:04.441 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.491 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd46d84a:::benchmark_data_smithi138_477915_object337:head# 2024-09-15T05:50:04.492 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.542 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd852d6c:::benchmark_data_smithi138_477915_object107:head# 2024-09-15T05:50:04.542 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.592 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd8a05cc:::benchmark_data_smithi138_477915_object2:head# 2024-09-15T05:50:04.592 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.642 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cec5b2bb:::benchmark_data_smithi138_477915_object25:head# 2024-09-15T05:50:04.643 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.726 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d0e38665:::benchmark_data_smithi138_477915_object266:head# 2024-09-15T05:50:04.726 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.793 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d165ae0f:::benchmark_data_smithi138_477915_object211:head# 2024-09-15T05:50:04.793 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.843 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d20fbe61:::benchmark_data_smithi138_477915_object284:head# 2024-09-15T05:50:04.843 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.910 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d350cfaa:::benchmark_data_smithi138_477915_object113:head# 2024-09-15T05:50:04.910 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:04.952 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d3b5e940:::benchmark_data_smithi138_477915_object14:head# 2024-09-15T05:50:04.952 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.002 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d48aafda:::benchmark_data_smithi138_477915_object358:head# 2024-09-15T05:50:05.002 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.053 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5a212a5:::benchmark_data_smithi138_477915_object106:head# 2024-09-15T05:50:05.053 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.094 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5d6c19c:::benchmark_data_smithi138_477915_object355:head# 2024-09-15T05:50:05.095 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.128 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6d89b5a:::benchmark_data_smithi138_477915_object341:head# 2024-09-15T05:50:05.128 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.162 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d789c5f7:::benchmark_data_smithi138_477915_object336:head# 2024-09-15T05:50:05.162 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.212 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c4ab31:::benchmark_data_smithi138_477915_object189:head# 2024-09-15T05:50:05.212 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.304 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c508f3:::benchmark_data_smithi138_477915_object4:head# 2024-09-15T05:50:05.304 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.371 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7ebcc2e:::benchmark_data_smithi138_477915_object265:head# 2024-09-15T05:50:05.372 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.430 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8daf539:::benchmark_data_smithi138_477915_object99:head# 2024-09-15T05:50:05.430 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.480 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8e85bad:::benchmark_data_smithi138_477915_object166:head# 2024-09-15T05:50:05.481 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.531 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d95dedd9:::benchmark_data_smithi138_477915_object274:head# 2024-09-15T05:50:05.531 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.581 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da2f71a7:::benchmark_data_smithi138_477915_object65:head# 2024-09-15T05:50:05.581 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.639 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da407726:::benchmark_data_smithi138_477915_object179:head# 2024-09-15T05:50:05.640 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.698 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dabfd9f7:::benchmark_data_smithi138_477915_object118:head# 2024-09-15T05:50:05.698 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.748 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dad5a76a:::benchmark_data_smithi138_477915_object305:head# 2024-09-15T05:50:05.748 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.798 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6aea31:::benchmark_data_smithi138_477915_object150:head# 2024-09-15T05:50:05.799 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd78a891:::benchmark_data_smithi138_477915_object353:head# 2024-09-15T05:50:05.849 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.899 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfac43cc:::benchmark_data_smithi138_477915_object126:head# 2024-09-15T05:50:05.899 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.949 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfe8aa68:::benchmark_data_smithi138_477915_object51:head# 2024-09-15T05:50:05.949 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:05.999 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e100c584:::benchmark_data_smithi138_477915_object8:head# 2024-09-15T05:50:06.000 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.050 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1828df5:::benchmark_data_smithi138_477915_object267:head# 2024-09-15T05:50:06.050 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.100 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1daa4ae:::benchmark_data_smithi138_477915_object37:head# 2024-09-15T05:50:06.100 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.150 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e23cd4da:::benchmark_data_smithi138_477915_object32:head# 2024-09-15T05:50:06.150 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.201 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4da1a27:::benchmark_data_smithi138_477915_object15:head# 2024-09-15T05:50:06.201 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.251 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4f1a68f:::benchmark_data_smithi138_477915_object395:head# 2024-09-15T05:50:06.251 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.301 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e6fdeca3:::benchmark_data_smithi138_477915_object338:head# 2024-09-15T05:50:06.301 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.351 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7c862ab:::benchmark_data_smithi138_477915_object119:head# 2024-09-15T05:50:06.351 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.401 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7e5da90:::benchmark_data_smithi138_477915_object393:head# 2024-09-15T05:50:06.401 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.452 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e9e1dfd6:::benchmark_data_smithi138_477915_object315:head# 2024-09-15T05:50:06.452 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.502 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eaee7ade:::benchmark_data_smithi138_477915_object277:head# 2024-09-15T05:50:06.502 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.544 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb2d0c81:::benchmark_data_smithi138_477915_object352:head# 2024-09-15T05:50:06.544 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.588 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb74e80f:::benchmark_data_smithi138_477915_object39:head# 2024-09-15T05:50:06.588 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.628 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ebd302e1:::benchmark_data_smithi138_477915_object221:head# 2024-09-15T05:50:06.628 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.670 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec96583a:::benchmark_data_smithi138_477915_object6:head# 2024-09-15T05:50:06.670 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecd13f7c:::benchmark_data_smithi138_477915_object174:head# 2024-09-15T05:50:06.745 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecda3af5:::benchmark_data_smithi138_477915_object7:head# 2024-09-15T05:50:06.787 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.829 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed2124e8:::benchmark_data_smithi138_477915_object283:head# 2024-09-15T05:50:06.829 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.870 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed4f8904:::benchmark_data_smithi138_477915_object321:head# 2024-09-15T05:50:06.871 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.912 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eec767c6:::benchmark_data_smithi138_477915_object304:head# 2024-09-15T05:50:06.913 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:06.963 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef20ff5b:::benchmark_data_smithi138_477915_object148:head# 2024-09-15T05:50:06.963 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.005 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef304116:::benchmark_data_smithi138_477915_object49:head# 2024-09-15T05:50:07.005 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.055 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef573daf:::benchmark_data_smithi138_477915_object177:head# 2024-09-15T05:50:07.055 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.105 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f0690f60:::benchmark_data_smithi138_477915_object397:head# 2024-09-15T05:50:07.106 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.156 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f15d302a:::benchmark_data_smithi138_477915_object81:head# 2024-09-15T05:50:07.156 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.239 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f2b2bc3f:::benchmark_data_smithi138_477915_object216:head# 2024-09-15T05:50:07.239 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.340 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f314e879:::benchmark_data_smithi138_477915_object290:head# 2024-09-15T05:50:07.340 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.410 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f358deca:::benchmark_data_smithi138_477915_object278:head# 2024-09-15T05:50:07.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.457 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f40ce7f6:::benchmark_data_smithi138_477915_object9:head# 2024-09-15T05:50:07.457 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f48dff42:::benchmark_data_smithi138_477915_object372:head# 2024-09-15T05:50:07.508 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.558 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4a59f6a:::benchmark_data_smithi138_477915_object12:head# 2024-09-15T05:50:07.558 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.608 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4d52c7d:::benchmark_data_smithi138_477915_object183:head# 2024-09-15T05:50:07.608 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.658 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f50e2ea1:::benchmark_data_smithi138_477915_object194:head# 2024-09-15T05:50:07.658 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.709 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5165677:::benchmark_data_smithi138_477915_object346:head# 2024-09-15T05:50:07.709 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.759 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5ba49b3:::benchmark_data_smithi138_477915_object333:head# 2024-09-15T05:50:07.759 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.809 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5c42dd1:::benchmark_data_smithi138_477915_object85:head# 2024-09-15T05:50:07.809 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.859 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f64f6100:::benchmark_data_smithi138_477915_object163:head# 2024-09-15T05:50:07.859 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.909 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f6daa659:::benchmark_data_smithi138_477915_object250:head# 2024-09-15T05:50:07.909 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:07.960 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f7f49d03:::benchmark_data_smithi138_477915_object324:head# 2024-09-15T05:50:07.960 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.010 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa26fae3:::benchmark_data_smithi138_477915_object73:head# 2024-09-15T05:50:08.010 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.060 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa6cc2ae:::benchmark_data_smithi138_477915_object91:head# 2024-09-15T05:50:08.060 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.110 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa9ab79f:::benchmark_data_smithi138_477915_object72:head# 2024-09-15T05:50:08.110 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.161 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fb6a8c6a:::benchmark_data_smithi138_477915_object122:head# 2024-09-15T05:50:08.161 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.203 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fba968a6:::benchmark_data_smithi138_477915_object368:head# 2024-09-15T05:50:08.203 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.244 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fd402ec6:::benchmark_data_smithi138_477915_object67:head# 2024-09-15T05:50:08.245 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.286 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fdef6d22:::benchmark_data_smithi138_477915_object309:head# 2024-09-15T05:50:08.286 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.328 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fe273b33:::benchmark_data_smithi138_477915_object146:head# 2024-09-15T05:50:08.328 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.370 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fea7c25b:::benchmark_data_smithi138_477915_object344:head# 2024-09-15T05:50:08.370 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.412 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ff1959bb:::benchmark_data_smithi138_477915_object80:head# 2024-09-15T05:50:08.412 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.454 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffb3fc0b:::benchmark_data_smithi138_477915_object115:head# 2024-09-15T05:50:08.454 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.496 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffbe43c0:::benchmark_data_smithi138_477915_object125:head# 2024-09-15T05:50:08.496 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:08.529 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 9 info 1.1( v 9'194 (0'0,9'194] local-lis/les=7/8 n=194 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:50:08.588 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:50:09.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-15T05:50:10.676 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:50:10.676 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:50:10.676 INFO:tasks.workunit.client.0.smithi138.stderr:PG export's map 9 is older than OSD's oldest_map 121 2024-09-15T05:50:10.677 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 2 -> 1 2024-09-15T05:50:10.677 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:004eb6a1:::benchmark_data_smithi138_477915_object24:head# 2024-09-15T05:50:10.677 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:10.851 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:01723c07:::benchmark_data_smithi138_477915_object335:head# 2024-09-15T05:50:10.851 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:10.901 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:021e9c3c:::benchmark_data_smithi138_477915_object60:head# 2024-09-15T05:50:10.901 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:10.951 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0276aebd:::benchmark_data_smithi138_477915_object108:head# 2024-09-15T05:50:10.951 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.001 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03123a40:::benchmark_data_smithi138_477915_object176:head# 2024-09-15T05:50:11.001 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.052 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:031b6711:::benchmark_data_smithi138_477915_object198:head# 2024-09-15T05:50:11.052 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.102 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0432e19c:::benchmark_data_smithi138_477915_object62:head# 2024-09-15T05:50:11.102 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.160 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04a09b67:::benchmark_data_smithi138_477915_object401:head# 2024-09-15T05:50:11.161 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.236 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04af6a20:::benchmark_data_smithi138_477915_object98:head# 2024-09-15T05:50:11.236 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.328 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05c5c04d:::benchmark_data_smithi138_477915_object195:head# 2024-09-15T05:50:11.328 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.386 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05ce29b1:::benchmark_data_smithi138_477915_object181:head# 2024-09-15T05:50:11.387 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05d45069:::benchmark_data_smithi138_477915_object275:head# 2024-09-15T05:50:11.437 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.487 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07351a71:::benchmark_data_smithi138_477915_object100:head# 2024-09-15T05:50:11.487 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.537 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07fbc7df:::benchmark_data_smithi138_477915_object312:head# 2024-09-15T05:50:11.537 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.579 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08079dbc:::benchmark_data_smithi138_477915_object299:head# 2024-09-15T05:50:11.579 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.630 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08859349:::benchmark_data_smithi138_477915_object223:head# 2024-09-15T05:50:11.630 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.672 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08a0ce86:::benchmark_data_smithi138_477915_object385:head# 2024-09-15T05:50:11.672 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.714 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08baf2ed:::benchmark_data_smithi138_477915_object192:head# 2024-09-15T05:50:11.714 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.764 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08bd9a02:::benchmark_data_smithi138_477915_object214:head# 2024-09-15T05:50:11.764 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:097fa44d:::benchmark_data_smithi138_477915_object328:head# 2024-09-15T05:50:11.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.848 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a0d4136:::benchmark_data_smithi138_477915_object30:head# 2024-09-15T05:50:11.848 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.906 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2641dd:::benchmark_data_smithi138_477915_object210:head# 2024-09-15T05:50:11.906 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:11.957 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2d6d17:::benchmark_data_smithi138_477915_object61:head# 2024-09-15T05:50:11.957 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.007 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ac84968:::benchmark_data_smithi138_477915_object286:head# 2024-09-15T05:50:12.007 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.057 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ad209ab:::benchmark_data_smithi138_477915_object70:head# 2024-09-15T05:50:12.057 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.124 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0b64715f:::benchmark_data_smithi138_477915_object17:head# 2024-09-15T05:50:12.124 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.174 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0bbf6561:::benchmark_data_smithi138_477915_object137:head# 2024-09-15T05:50:12.174 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.224 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0cca7b5b:::benchmark_data_smithi138_477915_object339:head# 2024-09-15T05:50:12.224 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.258 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0d194a0a:::benchmark_data_smithi138_477915_object77:head# 2024-09-15T05:50:12.258 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.291 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e135564:::benchmark_data_smithi138_477915_object313:head# 2024-09-15T05:50:12.291 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.325 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e15c428:::benchmark_data_smithi138_477915_object120:head# 2024-09-15T05:50:12.325 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.358 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0f4410e4:::benchmark_data_smithi138_477915_object145:head# 2024-09-15T05:50:12.358 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.391 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0fefd637:::benchmark_data_smithi138_477915_object389:head# 2024-09-15T05:50:12.392 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.425 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10535269:::benchmark_data_smithi138_477915_object332:head# 2024-09-15T05:50:12.425 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.458 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10bc54c8:::benchmark_data_smithi138_477915_object225:head# 2024-09-15T05:50:12.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.492 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10e6b441:::benchmark_data_smithi138_477915_object248:head# 2024-09-15T05:50:12.492 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.525 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1161a4f5:::benchmark_data_smithi138_477915_object281:head# 2024-09-15T05:50:12.525 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.592 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11ab1fa3:::benchmark_data_smithi138_477915_object240:head# 2024-09-15T05:50:12.593 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.634 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11b92b78:::benchmark_data_smithi138_477915_object142:head# 2024-09-15T05:50:12.634 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.667 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:13efdddb:::benchmark_data_smithi138_477915_object57:head# 2024-09-15T05:50:12.668 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.726 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14c54418:::benchmark_data_smithi138_477915_object197:head# 2024-09-15T05:50:12.726 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.776 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:152a39dd:::benchmark_data_smithi138_477915_object144:head# 2024-09-15T05:50:12.776 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.810 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15322a7a:::benchmark_data_smithi138_477915_object288:head# 2024-09-15T05:50:12.810 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.843 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15aaf075:::benchmark_data_smithi138_477915_object128:head# 2024-09-15T05:50:12.843 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.877 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16dad5c5:::benchmark_data_smithi138_477915_object252:head# 2024-09-15T05:50:12.877 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.910 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16f17ac6:::benchmark_data_smithi138_477915_object329:head# 2024-09-15T05:50:12.910 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.943 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1708f8e0:::benchmark_data_smithi138_477915_object307:head# 2024-09-15T05:50:12.944 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:12.977 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1854249c:::benchmark_data_smithi138_477915_object46:head# 2024-09-15T05:50:12.977 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.010 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:191dd67b:::benchmark_data_smithi138_477915_object20:head# 2024-09-15T05:50:13.011 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.044 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1acb70bd:::benchmark_data_smithi138_477915_object19:head# 2024-09-15T05:50:13.044 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.077 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b1b6402:::benchmark_data_smithi138_477915_object196:head# 2024-09-15T05:50:13.077 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.111 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b7cd855:::benchmark_data_smithi138_477915_object243:head# 2024-09-15T05:50:13.111 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.144 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1baea71e:::benchmark_data_smithi138_477915_object330:head# 2024-09-15T05:50:13.145 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.211 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1c0c4030:::benchmark_data_smithi138_477915_object325:head# 2024-09-15T05:50:13.211 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.261 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1ced30b9:::benchmark_data_smithi138_477915_object165:head# 2024-09-15T05:50:13.261 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.320 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dac8c2e:::benchmark_data_smithi138_477915_object204:head# 2024-09-15T05:50:13.320 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.370 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dc29afc:::benchmark_data_smithi138_477915_object398:head# 2024-09-15T05:50:13.370 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.404 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1f7be81b:::benchmark_data_smithi138_477915_object292:head# 2024-09-15T05:50:13.404 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20000fdb:::benchmark_data_smithi138_477915_object157:head# 2024-09-15T05:50:13.437 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.470 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20ce187a:::benchmark_data_smithi138_477915_object116:head# 2024-09-15T05:50:13.471 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.504 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:211ff0cd:::benchmark_data_smithi138_477915_object209:head# 2024-09-15T05:50:13.504 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.537 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:214fff42:::benchmark_data_smithi138_477915_object230:head# 2024-09-15T05:50:13.537 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.571 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:219197a2:::benchmark_data_smithi138_477915_object111:head# 2024-09-15T05:50:13.571 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.605 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:22101c95:::benchmark_data_smithi138_477915_object59:head# 2024-09-15T05:50:13.605 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.630 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:223f5228:::benchmark_data_smithi138_477915_object79:head# 2024-09-15T05:50:13.630 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.663 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:230de784:::benchmark_data_smithi138_477915_object244:head# 2024-09-15T05:50:13.663 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.688 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:246b4656:::benchmark_data_smithi138_477915_object191:head# 2024-09-15T05:50:13.689 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.713 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:25092a54:::benchmark_data_smithi138_477915_object10:head# 2024-09-15T05:50:13.714 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.739 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:263d2c7b:::benchmark_data_smithi138_477915_object153:head# 2024-09-15T05:50:13.739 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.764 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:266b4db5:::benchmark_data_smithi138_477915_object13:head# 2024-09-15T05:50:13.764 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.789 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26e68133:::benchmark_data_smithi138_477915_object349:head# 2024-09-15T05:50:13.789 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26f80018:::benchmark_data_smithi138_477915_object75:head# 2024-09-15T05:50:13.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.839 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2729df23:::benchmark_data_smithi138_477915_object245:head# 2024-09-15T05:50:13.839 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.864 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:273a39c8:::benchmark_data_smithi138_477915_object212:head# 2024-09-15T05:50:13.864 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.889 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b21cb84:::benchmark_data_smithi138_477915_object366:head# 2024-09-15T05:50:13.890 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.915 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2e0a6ba0:::benchmark_data_smithi138_477915_object45:head# 2024-09-15T05:50:13.915 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.940 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2fd3771a:::benchmark_data_smithi138_477915_object326:head# 2024-09-15T05:50:13.940 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.965 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30de888f:::benchmark_data_smithi138_477915_object34:head# 2024-09-15T05:50:13.965 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:13.990 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:312dc4e7:::benchmark_data_smithi138_477915_object232:head# 2024-09-15T05:50:13.990 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.015 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:318ba285:::benchmark_data_smithi138_477915_object164:head# 2024-09-15T05:50:14.015 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.040 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:31e1d00e:::benchmark_data_smithi138_477915_object44:head# 2024-09-15T05:50:14.040 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.065 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:32a36681:::benchmark_data_smithi138_477915_object90:head# 2024-09-15T05:50:14.065 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.090 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3397aa63:::benchmark_data_smithi138_477915_object294:head# 2024-09-15T05:50:14.091 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.115 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:350297ef:::benchmark_data_smithi138_477915_object382:head# 2024-09-15T05:50:14.116 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.141 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:354f77ae:::benchmark_data_smithi138_477915_object378:head# 2024-09-15T05:50:14.141 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.182 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:359485cb:::benchmark_data_smithi138_477915_object363:head# 2024-09-15T05:50:14.182 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.208 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:386dd901:::benchmark_data_smithi138_477915_object287:head# 2024-09-15T05:50:14.208 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.233 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:389afc62:::benchmark_data_smithi138_477915_object206:head# 2024-09-15T05:50:14.233 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.258 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38edce5f:::benchmark_data_smithi138_477915_object202:head# 2024-09-15T05:50:14.258 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.283 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:392023cd:::benchmark_data_smithi138_477915_object360:head# 2024-09-15T05:50:14.283 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.308 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:396a3af9:::benchmark_data_smithi138_477915_object361:head# 2024-09-15T05:50:14.308 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.333 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39870a18:::benchmark_data_smithi138_477915_object55:head# 2024-09-15T05:50:14.333 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.358 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a34f987:::benchmark_data_smithi138_477915_object84:head# 2024-09-15T05:50:14.358 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.383 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a45d3f0:::benchmark_data_smithi138_477915_object133:head# 2024-09-15T05:50:14.383 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.408 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc519ab:::benchmark_data_smithi138_477915_object301:head# 2024-09-15T05:50:14.409 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.433 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc57672:::benchmark_data_smithi138_477915_object317:head# 2024-09-15T05:50:14.433 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.467 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d290d02:::benchmark_data_smithi138_477915_object52:head# 2024-09-15T05:50:14.467 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.500 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d995ed5:::benchmark_data_smithi138_477915_object289:head# 2024-09-15T05:50:14.501 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.534 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3e073204:::benchmark_data_smithi138_477915_object47:head# 2024-09-15T05:50:14.534 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.567 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f24c379:::benchmark_data_smithi138_477915_object381:head# 2024-09-15T05:50:14.567 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.601 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f8ef9b0:::benchmark_data_smithi138_477915_object215:head# 2024-09-15T05:50:14.601 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.634 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4188c502:::benchmark_data_smithi138_477915_object394:head# 2024-09-15T05:50:14.634 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.668 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:41d0e70f:::benchmark_data_smithi138_477915_object124:head# 2024-09-15T05:50:14.668 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.709 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:42ce4ca2:::benchmark_data_smithi138_477915_object170:head# 2024-09-15T05:50:14.710 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.751 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4384dd5f:::benchmark_data_smithi138_477915_object222:head# 2024-09-15T05:50:14.751 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.785 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:441e81fc:::benchmark_data_smithi138_477915_object322:head# 2024-09-15T05:50:14.785 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.818 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:444f377d:::benchmark_data_smithi138_477915_object259:head# 2024-09-15T05:50:14.818 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.852 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44d1d1a6:::benchmark_data_smithi138_477915_object175:head# 2024-09-15T05:50:14.852 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.885 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44e95b23:::benchmark_data_smithi138_477915_object187:head# 2024-09-15T05:50:14.885 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.919 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44f216b2:::benchmark_data_smithi138_477915_object331:head# 2024-09-15T05:50:14.919 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.952 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45f59372:::benchmark_data_smithi138_477915_object134:head# 2024-09-15T05:50:14.952 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:14.986 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:463b99f0:::benchmark_data_smithi138_477915_object41:head# 2024-09-15T05:50:14.986 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.019 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:46f42845:::benchmark_data_smithi138_477915_object50:head# 2024-09-15T05:50:15.019 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.052 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:47847711:::benchmark_data_smithi138_477915_object391:head# 2024-09-15T05:50:15.053 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.086 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:481a4dcf:::benchmark_data_smithi138_477915_object306:head# 2024-09-15T05:50:15.086 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.119 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48339ebf:::benchmark_data_smithi138_477915_object53:head# 2024-09-15T05:50:15.119 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.153 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48cd125a:::benchmark_data_smithi138_477915_object320:head# 2024-09-15T05:50:15.153 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.189 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:49297529:::benchmark_data_smithi138_477915_object110:head# 2024-09-15T05:50:15.192 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.237 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:492bb000:::benchmark_data_smithi138_477915_object364:head# 2024-09-15T05:50:15.237 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.295 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4a0a3214:::benchmark_data_smithi138_477915_object220:head# 2024-09-15T05:50:15.296 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.345 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4b9543c5:::benchmark_data_smithi138_477915_object1:head# 2024-09-15T05:50:15.346 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.379 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba0927c:::benchmark_data_smithi138_477915_object178:head# 2024-09-15T05:50:15.379 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.429 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba2f324:::benchmark_data_smithi138_477915_object293:head# 2024-09-15T05:50:15.429 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.463 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c0b6957:::benchmark_data_smithi138_477915_object114:head# 2024-09-15T05:50:15.463 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.496 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c7779cc:::benchmark_data_smithi138_477915_object379:head# 2024-09-15T05:50:15.497 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.530 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4cffde3a:::benchmark_data_smithi138_477915_object271:head# 2024-09-15T05:50:15.530 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.563 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d3925ed:::benchmark_data_smithi138_477915_object396:head# 2024-09-15T05:50:15.563 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.597 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d67e2bb:::benchmark_data_smithi138_477915_object141:head# 2024-09-15T05:50:15.597 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.630 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4e93ce33:::benchmark_data_smithi138_477915_object82:head# 2024-09-15T05:50:15.630 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.689 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4eb39cc3:::benchmark_data_smithi138_477915_object11:head# 2024-09-15T05:50:15.689 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.714 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4fdb5ea1:::benchmark_data_smithi138_477915_object86:head# 2024-09-15T05:50:15.714 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.739 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52b28e51:::benchmark_data_smithi138_477915_object188:head# 2024-09-15T05:50:15.739 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.764 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52f1d98b:::benchmark_data_smithi138_477915_object29:head# 2024-09-15T05:50:15.765 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.789 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:538d02e4:::benchmark_data_smithi138_477915_object387:head# 2024-09-15T05:50:15.790 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.815 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:53b28f04:::benchmark_data_smithi138_477915_object193:head# 2024-09-15T05:50:15.815 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.839 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:543e73a1:::benchmark_data_smithi138_477915_object143:head# 2024-09-15T05:50:15.840 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.873 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:546413ea:::benchmark_data_smithi138_477915_object35:head# 2024-09-15T05:50:15.873 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.940 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54b5d110:::benchmark_data_smithi138_477915_object370:head# 2024-09-15T05:50:15.940 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.965 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54d808a1:::benchmark_data_smithi138_477915_object147:head# 2024-09-15T05:50:15.965 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:15.990 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:556f515c:::benchmark_data_smithi138_477915_object367:head# 2024-09-15T05:50:15.990 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.017 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5581a3b9:::benchmark_data_smithi138_477915_object123:head# 2024-09-15T05:50:16.017 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.040 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55b4ce8c:::benchmark_data_smithi138_477915_object327:head# 2024-09-15T05:50:16.040 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.066 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55e0f391:::benchmark_data_smithi138_477915_object40:head# 2024-09-15T05:50:16.066 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.091 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:564dfa9a:::benchmark_data_smithi138_477915_object89:head# 2024-09-15T05:50:16.091 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.116 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56619472:::benchmark_data_smithi138_477915_object291:head# 2024-09-15T05:50:16.116 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.141 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56ac2e18:::benchmark_data_smithi138_477915_object373:head# 2024-09-15T05:50:16.141 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.166 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e0ff0d:::benchmark_data_smithi138_477915_object246:head# 2024-09-15T05:50:16.166 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.191 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e54ad7:::benchmark_data_smithi138_477915_object241:head# 2024-09-15T05:50:16.192 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.216 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:572a26cc:::benchmark_data_smithi138_477915_object3:head# 2024-09-15T05:50:16.216 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.241 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:57365c69:::benchmark_data_smithi138_477915_object185:head# 2024-09-15T05:50:16.242 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.266 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:58b05a98:::benchmark_data_smithi138_477915_object94:head# 2024-09-15T05:50:16.267 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.292 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5979f860:::benchmark_data_smithi138_477915_object42:head# 2024-09-15T05:50:16.292 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.317 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5a3930c3:::benchmark_data_smithi138_477915_object92:head# 2024-09-15T05:50:16.317 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.342 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5acdf762:::benchmark_data_smithi138_477915_object217:head# 2024-09-15T05:50:16.342 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.367 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5b192008:::benchmark_data_smithi138_477915_object190:head# 2024-09-15T05:50:16.367 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.434 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5bd647c9:::benchmark_data_smithi138_477915_object308:head# 2024-09-15T05:50:16.434 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c5abf37:::benchmark_data_smithi138_477915_object227:head# 2024-09-15T05:50:16.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.484 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5cc75e94:::benchmark_data_smithi138_477915_object261:head# 2024-09-15T05:50:16.484 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.509 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5d452870:::benchmark_data_smithi138_477915_object386:head# 2024-09-15T05:50:16.509 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.534 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5e2e9211:::benchmark_data_smithi138_477915_object201:head# 2024-09-15T05:50:16.534 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.560 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5f10ef7e:::benchmark_data_smithi138_477915_object182:head# 2024-09-15T05:50:16.560 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.584 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fa8bc7a:::benchmark_data_smithi138_477915_object371:head# 2024-09-15T05:50:16.584 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.610 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fd3194b:::benchmark_data_smithi138_477915_object83:head# 2024-09-15T05:50:16.610 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.635 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fec7540:::benchmark_data_smithi138_477915_object132:head# 2024-09-15T05:50:16.635 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.660 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6012bbb1:::benchmark_data_smithi138_477915_object213:head# 2024-09-15T05:50:16.661 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.710 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:620660f2:::benchmark_data_smithi138_477915_object138:head# 2024-09-15T05:50:16.710 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.752 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:62fcea3a:::benchmark_data_smithi138_477915_object374:head# 2024-09-15T05:50:16.752 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.785 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:635fcc02:::benchmark_data_smithi138_477915_object18:head# 2024-09-15T05:50:16.785 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.819 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6429912b:::benchmark_data_smithi138_477915_object78:head# 2024-09-15T05:50:16.819 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.852 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:648cc3c2:::benchmark_data_smithi138_477915_object180:head# 2024-09-15T05:50:16.853 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.894 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64ae8f93:::benchmark_data_smithi138_477915_object273:head# 2024-09-15T05:50:16.895 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.928 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:65851d9e:::benchmark_data_smithi138_477915_object276:head# 2024-09-15T05:50:16.928 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.961 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:66acc5ee:::benchmark_data_smithi138_477915_object155:head# 2024-09-15T05:50:16.961 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:16.994 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:67e15da6:::benchmark_data_smithi138_477915_object253:head# 2024-09-15T05:50:16.995 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.028 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68a5ecca:::benchmark_data_smithi138_477915_object260:head# 2024-09-15T05:50:17.028 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.061 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68b33428:::benchmark_data_smithi138_477915_object388:head# 2024-09-15T05:50:17.062 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.095 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6940b9d3:::benchmark_data_smithi138_477915_object239:head# 2024-09-15T05:50:17.095 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.128 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:69f7ef4c:::benchmark_data_smithi138_477915_object127:head# 2024-09-15T05:50:17.129 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.162 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6aef21a2:::benchmark_data_smithi138_477915_object54:head# 2024-09-15T05:50:17.162 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.206 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6b3c2b63:::benchmark_data_smithi138_477915_object208:head# 2024-09-15T05:50:17.206 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.263 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bf4848f:::benchmark_data_smithi138_477915_object399:head# 2024-09-15T05:50:17.263 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.313 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ccf34bf:::benchmark_data_smithi138_477915_object22:head# 2024-09-15T05:50:17.313 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.371 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6d5a3232:::benchmark_data_smithi138_477915_object43:head# 2024-09-15T05:50:17.372 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.405 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dc7f41b:::benchmark_data_smithi138_477915_object262:head# 2024-09-15T05:50:17.405 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.438 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dd17fb9:::benchmark_data_smithi138_477915_object263:head# 2024-09-15T05:50:17.438 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.472 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ddcb643:::benchmark_data_smithi138_477915_object285:head# 2024-09-15T05:50:17.472 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.505 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6e1716aa:::benchmark_data_smithi138_477915_object135:head# 2024-09-15T05:50:17.505 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.539 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:702ed571:::benchmark_data_smithi138_477915_object219:head# 2024-09-15T05:50:17.539 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.572 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:707bb828:::benchmark_data_smithi138_477915_object158:head# 2024-09-15T05:50:17.572 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.622 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:70ca4348:::benchmark_data_smithi138_477915_object26:head# 2024-09-15T05:50:17.622 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.656 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71784a9a:::benchmark_data_smithi138_477915_object376:head# 2024-09-15T05:50:17.656 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.689 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:717dd36a:::benchmark_data_smithi138_477915_object139:head# 2024-09-15T05:50:17.689 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.723 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71fb5440:::benchmark_data_smithi138_477915_object97:head# 2024-09-15T05:50:17.723 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.781 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7314c8d4:::benchmark_data_smithi138_477915_object152:head# 2024-09-15T05:50:17.782 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:740159d0:::benchmark_data_smithi138_477915_object0:head# 2024-09-15T05:50:17.815 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.848 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:74886e53:::benchmark_data_smithi138_477915_object112:head# 2024-09-15T05:50:17.848 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.882 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:75046150:::benchmark_data_smithi138_477915_object279:head# 2024-09-15T05:50:17.882 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.915 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:773e28fe:::benchmark_data_smithi138_477915_object377:head# 2024-09-15T05:50:17.916 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.948 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7783b60c:::benchmark_data_smithi138_477915_object101:head# 2024-09-15T05:50:17.949 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.973 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:786160e4:::benchmark_data_smithi138_477915_object38:head# 2024-09-15T05:50:17.974 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:17.999 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a2bef6b:::benchmark_data_smithi138_477915_object58:head# 2024-09-15T05:50:17.999 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.024 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a76feec:::benchmark_data_smithi138_477915_object63:head# 2024-09-15T05:50:18.024 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.049 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7bb84b6a:::benchmark_data_smithi138_477915_object5:head# 2024-09-15T05:50:18.049 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.074 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7be49943:::benchmark_data_smithi138_477915_object48:head# 2024-09-15T05:50:18.074 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.099 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d976229:::benchmark_data_smithi138_477915_object207:head# 2024-09-15T05:50:18.099 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.149 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d9d7a5f:::benchmark_data_smithi138_477915_object369:head# 2024-09-15T05:50:18.150 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.191 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eaec87b:::benchmark_data_smithi138_477915_object154:head# 2024-09-15T05:50:18.191 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.242 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eda30d0:::benchmark_data_smithi138_477915_object310:head# 2024-09-15T05:50:18.242 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.292 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7f744066:::benchmark_data_smithi138_477915_object205:head# 2024-09-15T05:50:18.292 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:18.342 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 9 info 1.0( v 9'209 (0'0,9'209] local-lis/les=7/8 n=209 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:50:18.392 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:50:18.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-15T05:50:20.582 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:50:20.582 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:50:20.582 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T05:50:21.006 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:50:21.351 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-15T05:50:23.042 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:50:23.042 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:50:23.061 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-15T05:50:23.673 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:50:24.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-09-15T05:50:25.694 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-09-15T05:50:25.694 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:50:25.694 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:004eb6a1:::benchmark_data_smithi138_477915_object24:head# 2024-09-15T05:50:25.694 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:25.802 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:005bff5c:::benchmark_data_smithi138_483817_object135:head# 2024-09-15T05:50:25.802 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:25.852 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0149ad20:::benchmark_data_smithi138_483817_object63:head# 2024-09-15T05:50:25.852 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:25.902 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:01723c07:::benchmark_data_smithi138_477915_object335:head# 2024-09-15T05:50:25.903 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:25.953 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:021e9c3c:::benchmark_data_smithi138_477915_object60:head# 2024-09-15T05:50:25.953 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:25.978 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0255b854:::benchmark_data_smithi138_483817_object75:head# 2024-09-15T05:50:25.978 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.003 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0276aebd:::benchmark_data_smithi138_477915_object108:head# 2024-09-15T05:50:26.003 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.028 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03123a40:::benchmark_data_smithi138_477915_object176:head# 2024-09-15T05:50:26.028 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.053 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:031b6711:::benchmark_data_smithi138_477915_object198:head# 2024-09-15T05:50:26.054 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.078 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03a96464:::benchmark_data_smithi138_483817_object19:head# 2024-09-15T05:50:26.079 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.103 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0432e19c:::benchmark_data_smithi138_477915_object62:head# 2024-09-15T05:50:26.103 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.129 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0465873b:::benchmark_data_smithi138_483817_object34:head# 2024-09-15T05:50:26.129 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.156 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04a09b67:::benchmark_data_smithi138_477915_object401:head# 2024-09-15T05:50:26.156 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.189 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:04af6a20:::benchmark_data_smithi138_477915_object98:head# 2024-09-15T05:50:26.190 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.223 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05c5c04d:::benchmark_data_smithi138_477915_object195:head# 2024-09-15T05:50:26.223 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.256 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05ce29b1:::benchmark_data_smithi138_477915_object181:head# 2024-09-15T05:50:26.257 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.290 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05d45069:::benchmark_data_smithi138_477915_object275:head# 2024-09-15T05:50:26.290 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.323 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07351a71:::benchmark_data_smithi138_477915_object100:head# 2024-09-15T05:50:26.323 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.357 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:07fbc7df:::benchmark_data_smithi138_477915_object312:head# 2024-09-15T05:50:26.357 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.407 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08079dbc:::benchmark_data_smithi138_477915_object299:head# 2024-09-15T05:50:26.407 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.440 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08859349:::benchmark_data_smithi138_477915_object223:head# 2024-09-15T05:50:26.440 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.474 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08a0ce86:::benchmark_data_smithi138_477915_object385:head# 2024-09-15T05:50:26.474 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08baf2ed:::benchmark_data_smithi138_477915_object192:head# 2024-09-15T05:50:26.507 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.540 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08bd9a02:::benchmark_data_smithi138_477915_object214:head# 2024-09-15T05:50:26.541 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.574 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0973fe62:::benchmark_data_smithi138_483817_object92:head# 2024-09-15T05:50:26.574 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.607 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:097fa44d:::benchmark_data_smithi138_477915_object328:head# 2024-09-15T05:50:26.608 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.641 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:09e06460:::benchmark_data_smithi138_483817_object133:head# 2024-09-15T05:50:26.641 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.684 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a0d4136:::benchmark_data_smithi138_477915_object30:head# 2024-09-15T05:50:26.684 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.733 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2641dd:::benchmark_data_smithi138_477915_object210:head# 2024-09-15T05:50:26.733 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.767 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a2d6d17:::benchmark_data_smithi138_477915_object61:head# 2024-09-15T05:50:26.767 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.800 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ac84968:::benchmark_data_smithi138_477915_object286:head# 2024-09-15T05:50:26.800 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.834 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ad209ab:::benchmark_data_smithi138_477915_object70:head# 2024-09-15T05:50:26.834 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.867 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0af91a19:::benchmark_data_smithi138_483817_object15:head# 2024-09-15T05:50:26.867 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.892 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0b64715f:::benchmark_data_smithi138_477915_object17:head# 2024-09-15T05:50:26.893 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.917 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ba493f4:::benchmark_data_smithi138_483817_object86:head# 2024-09-15T05:50:26.917 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.942 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0bbf6561:::benchmark_data_smithi138_477915_object137:head# 2024-09-15T05:50:26.942 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.967 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0cca7b5b:::benchmark_data_smithi138_477915_object339:head# 2024-09-15T05:50:26.968 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:26.993 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0cd0de56:::benchmark_data_smithi138_483817_object122:head# 2024-09-15T05:50:26.993 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.018 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0d194a0a:::benchmark_data_smithi138_477915_object77:head# 2024-09-15T05:50:27.018 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.043 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e09c97e:::benchmark_data_smithi138_483817_object82:head# 2024-09-15T05:50:27.043 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.068 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e135564:::benchmark_data_smithi138_477915_object313:head# 2024-09-15T05:50:27.068 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.093 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0e15c428:::benchmark_data_smithi138_477915_object120:head# 2024-09-15T05:50:27.093 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.118 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0f4410e4:::benchmark_data_smithi138_477915_object145:head# 2024-09-15T05:50:27.118 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.143 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0f9a013a:::benchmark_data_smithi138_483817_object76:head# 2024-09-15T05:50:27.143 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.168 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0fefd637:::benchmark_data_smithi138_477915_object389:head# 2024-09-15T05:50:27.169 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.202 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10535269:::benchmark_data_smithi138_477915_object332:head# 2024-09-15T05:50:27.202 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.252 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10796dab:::benchmark_data_smithi138_483817_object99:head# 2024-09-15T05:50:27.253 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.303 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10bc54c8:::benchmark_data_smithi138_477915_object225:head# 2024-09-15T05:50:27.303 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.353 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10e6b441:::benchmark_data_smithi138_477915_object248:head# 2024-09-15T05:50:27.353 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.378 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:111a5576:::benchmark_data_smithi138_483817_object124:head# 2024-09-15T05:50:27.378 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.412 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1161a4f5:::benchmark_data_smithi138_477915_object281:head# 2024-09-15T05:50:27.412 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11821dac:::benchmark_data_smithi138_483817_object62:head# 2024-09-15T05:50:27.437 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.462 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11ab1fa3:::benchmark_data_smithi138_477915_object240:head# 2024-09-15T05:50:27.462 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.487 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:11b92b78:::benchmark_data_smithi138_477915_object142:head# 2024-09-15T05:50:27.487 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.512 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:13efdddb:::benchmark_data_smithi138_477915_object57:head# 2024-09-15T05:50:27.512 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.537 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14c54418:::benchmark_data_smithi138_477915_object197:head# 2024-09-15T05:50:27.537 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.562 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:152a39dd:::benchmark_data_smithi138_477915_object144:head# 2024-09-15T05:50:27.562 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.587 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15322a7a:::benchmark_data_smithi138_477915_object288:head# 2024-09-15T05:50:27.588 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.613 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15aaf075:::benchmark_data_smithi138_477915_object128:head# 2024-09-15T05:50:27.613 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.638 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16a80714:::benchmark_data_smithi138_483817_object56:head# 2024-09-15T05:50:27.638 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.663 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16d2ffca:::benchmark_data_smithi138_483817_object168:head# 2024-09-15T05:50:27.663 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.688 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16dad5c5:::benchmark_data_smithi138_477915_object252:head# 2024-09-15T05:50:27.688 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.713 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16f17ac6:::benchmark_data_smithi138_477915_object329:head# 2024-09-15T05:50:27.713 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.738 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1708f8e0:::benchmark_data_smithi138_477915_object307:head# 2024-09-15T05:50:27.738 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.763 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1854249c:::benchmark_data_smithi138_477915_object46:head# 2024-09-15T05:50:27.763 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.805 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:18ee0375:::benchmark_data_smithi138_483817_object18:head# 2024-09-15T05:50:27.805 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.830 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:19010e40:::benchmark_data_smithi138_483817_object96:head# 2024-09-15T05:50:27.830 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.855 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:191dd67b:::benchmark_data_smithi138_477915_object20:head# 2024-09-15T05:50:27.856 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.914 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1a10802d:::benchmark_data_smithi138_483817_object11:head# 2024-09-15T05:50:27.914 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.964 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1acb70bd:::benchmark_data_smithi138_477915_object19:head# 2024-09-15T05:50:27.965 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:27.998 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b1b6402:::benchmark_data_smithi138_477915_object196:head# 2024-09-15T05:50:27.998 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.031 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b49f001:::benchmark_data_smithi138_483817_object77:head# 2024-09-15T05:50:28.031 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.065 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b7cd855:::benchmark_data_smithi138_477915_object243:head# 2024-09-15T05:50:28.065 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.098 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1baea71e:::benchmark_data_smithi138_477915_object330:head# 2024-09-15T05:50:28.098 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.132 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1c0c4030:::benchmark_data_smithi138_477915_object325:head# 2024-09-15T05:50:28.132 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.165 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1ced30b9:::benchmark_data_smithi138_477915_object165:head# 2024-09-15T05:50:28.165 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.199 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dac8c2e:::benchmark_data_smithi138_477915_object204:head# 2024-09-15T05:50:28.199 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.232 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1dc29afc:::benchmark_data_smithi138_477915_object398:head# 2024-09-15T05:50:28.232 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.265 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1f7be81b:::benchmark_data_smithi138_477915_object292:head# 2024-09-15T05:50:28.266 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.299 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1f84ed61:::benchmark_data_smithi138_483817_object172:head# 2024-09-15T05:50:28.299 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.332 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20000fdb:::benchmark_data_smithi138_477915_object157:head# 2024-09-15T05:50:28.332 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.366 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20909dac:::benchmark_data_smithi138_483817_object83:head# 2024-09-15T05:50:28.366 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.399 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20ce187a:::benchmark_data_smithi138_477915_object116:head# 2024-09-15T05:50:28.399 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.433 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:211ff0cd:::benchmark_data_smithi138_477915_object209:head# 2024-09-15T05:50:28.433 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.483 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:214fff42:::benchmark_data_smithi138_477915_object230:head# 2024-09-15T05:50:28.483 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.516 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:219197a2:::benchmark_data_smithi138_477915_object111:head# 2024-09-15T05:50:28.517 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.550 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:21be39a5:::benchmark_data_smithi138_483817_object31:head# 2024-09-15T05:50:28.550 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.583 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:22101c95:::benchmark_data_smithi138_477915_object59:head# 2024-09-15T05:50:28.583 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.617 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:223f5228:::benchmark_data_smithi138_477915_object79:head# 2024-09-15T05:50:28.617 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.650 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:22df1d28:::benchmark_data_smithi138_483817_object64:head# 2024-09-15T05:50:28.650 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.692 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:230de784:::benchmark_data_smithi138_477915_object244:head# 2024-09-15T05:50:28.692 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.742 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:246b4656:::benchmark_data_smithi138_477915_object191:head# 2024-09-15T05:50:28.743 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.793 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:25092a54:::benchmark_data_smithi138_477915_object10:head# 2024-09-15T05:50:28.793 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.826 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:251f2935:::benchmark_data_smithi138_483817_object49:head# 2024-09-15T05:50:28.826 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.859 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:263d2c7b:::benchmark_data_smithi138_477915_object153:head# 2024-09-15T05:50:28.860 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.893 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:266b4db5:::benchmark_data_smithi138_477915_object13:head# 2024-09-15T05:50:28.893 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.926 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26e68133:::benchmark_data_smithi138_477915_object349:head# 2024-09-15T05:50:28.927 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.951 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26f80018:::benchmark_data_smithi138_477915_object75:head# 2024-09-15T05:50:28.952 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:28.977 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2714c0b4:::benchmark_data_smithi138_483817_object156:head# 2024-09-15T05:50:28.977 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.002 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2729df23:::benchmark_data_smithi138_477915_object245:head# 2024-09-15T05:50:29.002 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.027 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:273a39c8:::benchmark_data_smithi138_477915_object212:head# 2024-09-15T05:50:29.027 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.052 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:287d6080:::benchmark_data_smithi138_483817_object70:head# 2024-09-15T05:50:29.052 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.077 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:29dae954:::benchmark_data_smithi138_483817_object117:head# 2024-09-15T05:50:29.077 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.104 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2a1a585d:::benchmark_data_smithi138_483817_object38:head# 2024-09-15T05:50:29.105 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.127 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2a9526df:::benchmark_data_smithi138_483817_object153:head# 2024-09-15T05:50:29.127 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.153 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2a962538:::benchmark_data_smithi138_483817_object101:head# 2024-09-15T05:50:29.153 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.178 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b0c02cc:::benchmark_data_smithi138_483817_object2:head# 2024-09-15T05:50:29.178 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.253 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b21cb84:::benchmark_data_smithi138_477915_object366:head# 2024-09-15T05:50:29.253 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.303 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b8191b4:::benchmark_data_smithi138_483817_object113:head# 2024-09-15T05:50:29.304 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.362 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b97e84e:::benchmark_data_smithi138_483817_object142:head# 2024-09-15T05:50:29.362 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.387 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2ba68ae5:::benchmark_data_smithi138_483817_object109:head# 2024-09-15T05:50:29.387 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.429 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2e0a6ba0:::benchmark_data_smithi138_477915_object45:head# 2024-09-15T05:50:29.429 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.454 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2ea804a3:::benchmark_data_smithi138_483817_object13:head# 2024-09-15T05:50:29.454 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.479 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2ecabff5:::benchmark_data_smithi138_483817_object106:head# 2024-09-15T05:50:29.479 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.505 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2eddd7ca:::benchmark_data_smithi138_483817_object129:head# 2024-09-15T05:50:29.505 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.530 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2f2263ed:::benchmark_data_smithi138_483817_object43:head# 2024-09-15T05:50:29.530 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.555 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2f2ab72c:::benchmark_data_smithi138_483817_object60:head# 2024-09-15T05:50:29.555 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.580 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2fd3771a:::benchmark_data_smithi138_477915_object326:head# 2024-09-15T05:50:29.580 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.605 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3097204c:::benchmark_data_smithi138_483817_object131:head# 2024-09-15T05:50:29.605 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.630 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30b22779:::benchmark_data_smithi138_483817_object128:head# 2024-09-15T05:50:29.630 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.655 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30de888f:::benchmark_data_smithi138_477915_object34:head# 2024-09-15T05:50:29.655 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.680 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:312dc4e7:::benchmark_data_smithi138_477915_object232:head# 2024-09-15T05:50:29.681 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.705 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:318ba285:::benchmark_data_smithi138_477915_object164:head# 2024-09-15T05:50:29.705 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.739 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:31e1d00e:::benchmark_data_smithi138_477915_object44:head# 2024-09-15T05:50:29.739 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.764 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:32a36681:::benchmark_data_smithi138_477915_object90:head# 2024-09-15T05:50:29.764 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.789 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3397aa63:::benchmark_data_smithi138_477915_object294:head# 2024-09-15T05:50:29.790 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.815 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:33bfd532:::benchmark_data_smithi138_483817_object176:head# 2024-09-15T05:50:29.815 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.840 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:350297ef:::benchmark_data_smithi138_477915_object382:head# 2024-09-15T05:50:29.840 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.881 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:35469d69:::benchmark_data_smithi138_483817_object132:head# 2024-09-15T05:50:29.882 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.923 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:354f77ae:::benchmark_data_smithi138_477915_object378:head# 2024-09-15T05:50:29.924 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.949 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:359485cb:::benchmark_data_smithi138_477915_object363:head# 2024-09-15T05:50:29.949 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:29.982 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:386459e6:::benchmark_data_smithi138_483817_object152:head# 2024-09-15T05:50:29.982 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.015 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:386dd901:::benchmark_data_smithi138_477915_object287:head# 2024-09-15T05:50:30.016 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.049 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:389afc62:::benchmark_data_smithi138_477915_object206:head# 2024-09-15T05:50:30.049 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.082 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38cb0823:::benchmark_data_smithi138_483817_object54:head# 2024-09-15T05:50:30.082 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.116 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38edce5f:::benchmark_data_smithi138_477915_object202:head# 2024-09-15T05:50:30.116 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.158 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:392023cd:::benchmark_data_smithi138_477915_object360:head# 2024-09-15T05:50:30.158 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.199 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3959ef6d:::benchmark_data_smithi138_483817_object66:head# 2024-09-15T05:50:30.200 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.233 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:396a3af9:::benchmark_data_smithi138_477915_object361:head# 2024-09-15T05:50:30.233 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.266 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39870a18:::benchmark_data_smithi138_477915_object55:head# 2024-09-15T05:50:30.266 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.300 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39d4ba45:::benchmark_data_smithi138_483817_object107:head# 2024-09-15T05:50:30.300 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.333 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a34f987:::benchmark_data_smithi138_477915_object84:head# 2024-09-15T05:50:30.333 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.367 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a45d3f0:::benchmark_data_smithi138_477915_object133:head# 2024-09-15T05:50:30.367 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.425 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc519ab:::benchmark_data_smithi138_477915_object301:head# 2024-09-15T05:50:30.425 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3bc57672:::benchmark_data_smithi138_477915_object317:head# 2024-09-15T05:50:30.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.492 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d290d02:::benchmark_data_smithi138_477915_object52:head# 2024-09-15T05:50:30.492 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.526 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d995ed5:::benchmark_data_smithi138_477915_object289:head# 2024-09-15T05:50:30.526 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.559 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3deada5a:::benchmark_data_smithi138_483817_object69:head# 2024-09-15T05:50:30.559 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.593 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3e073204:::benchmark_data_smithi138_477915_object47:head# 2024-09-15T05:50:30.593 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.626 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3eb24b0a:::benchmark_data_smithi138_483817_object127:head# 2024-09-15T05:50:30.626 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.659 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f24c379:::benchmark_data_smithi138_477915_object381:head# 2024-09-15T05:50:30.659 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.726 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f8ef9b0:::benchmark_data_smithi138_477915_object215:head# 2024-09-15T05:50:30.727 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.777 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4188c502:::benchmark_data_smithi138_477915_object394:head# 2024-09-15T05:50:30.777 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.810 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:41d0e70f:::benchmark_data_smithi138_477915_object124:head# 2024-09-15T05:50:30.810 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.843 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:42ce4ca2:::benchmark_data_smithi138_477915_object170:head# 2024-09-15T05:50:30.844 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.877 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4384dd5f:::benchmark_data_smithi138_477915_object222:head# 2024-09-15T05:50:30.877 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.919 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:43888adc:::benchmark_data_smithi138_483817_object137:head# 2024-09-15T05:50:30.919 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.953 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:441e81fc:::benchmark_data_smithi138_477915_object322:head# 2024-09-15T05:50:30.953 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:30.986 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:444f377d:::benchmark_data_smithi138_477915_object259:head# 2024-09-15T05:50:30.986 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.019 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44849f0f:::benchmark_data_smithi138_483817_object10:head# 2024-09-15T05:50:31.019 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.053 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:448c8ce2:::benchmark_data_smithi138_483817_object175:head# 2024-09-15T05:50:31.053 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.078 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44b71f94:::benchmark_data_smithi138_483817_object130:head# 2024-09-15T05:50:31.078 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.103 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44d1d1a6:::benchmark_data_smithi138_477915_object175:head# 2024-09-15T05:50:31.103 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.136 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44e95b23:::benchmark_data_smithi138_477915_object187:head# 2024-09-15T05:50:31.137 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.162 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44f216b2:::benchmark_data_smithi138_477915_object331:head# 2024-09-15T05:50:31.162 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.186 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45612842:::benchmark_data_smithi138_483817_object6:head# 2024-09-15T05:50:31.187 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.237 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45f59372:::benchmark_data_smithi138_477915_object134:head# 2024-09-15T05:50:31.237 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.279 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:463b99f0:::benchmark_data_smithi138_477915_object41:head# 2024-09-15T05:50:31.279 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.329 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4653fe80:::benchmark_data_smithi138_483817_object29:head# 2024-09-15T05:50:31.329 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.354 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:46e620bd:::benchmark_data_smithi138_483817_object42:head# 2024-09-15T05:50:31.354 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.380 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:46f42845:::benchmark_data_smithi138_477915_object50:head# 2024-09-15T05:50:31.380 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.405 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:47847711:::benchmark_data_smithi138_477915_object391:head# 2024-09-15T05:50:31.405 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.430 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:47e9ebca:::benchmark_data_smithi138_483817_object35:head# 2024-09-15T05:50:31.430 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.455 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:481a4dcf:::benchmark_data_smithi138_477915_object306:head# 2024-09-15T05:50:31.455 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.480 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48339ebf:::benchmark_data_smithi138_477915_object53:head# 2024-09-15T05:50:31.480 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.505 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48cd125a:::benchmark_data_smithi138_477915_object320:head# 2024-09-15T05:50:31.505 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.530 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:49297529:::benchmark_data_smithi138_477915_object110:head# 2024-09-15T05:50:31.530 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.555 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:492bb000:::benchmark_data_smithi138_477915_object364:head# 2024-09-15T05:50:31.556 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.581 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:49cc79d0:::benchmark_data_smithi138_483817_object67:head# 2024-09-15T05:50:31.582 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.606 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4a0a3214:::benchmark_data_smithi138_477915_object220:head# 2024-09-15T05:50:31.606 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.631 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4b9543c5:::benchmark_data_smithi138_477915_object1:head# 2024-09-15T05:50:31.631 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.641 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba0927c:::benchmark_data_smithi138_477915_object178:head# 2024-09-15T05:50:31.641 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.664 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4ba2f324:::benchmark_data_smithi138_477915_object293:head# 2024-09-15T05:50:31.665 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.706 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c0b6957:::benchmark_data_smithi138_477915_object114:head# 2024-09-15T05:50:31.706 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.759 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c11c9f7:::benchmark_data_smithi138_483817_object9:head# 2024-09-15T05:50:31.759 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.790 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4c7779cc:::benchmark_data_smithi138_477915_object379:head# 2024-09-15T05:50:31.790 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.815 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4cb16b08:::benchmark_data_smithi138_483817_object37:head# 2024-09-15T05:50:31.815 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.840 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4cffde3a:::benchmark_data_smithi138_477915_object271:head# 2024-09-15T05:50:31.840 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.865 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d3925ed:::benchmark_data_smithi138_477915_object396:head# 2024-09-15T05:50:31.865 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.890 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4d67e2bb:::benchmark_data_smithi138_477915_object141:head# 2024-09-15T05:50:31.891 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.915 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4e93ce33:::benchmark_data_smithi138_477915_object82:head# 2024-09-15T05:50:31.916 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.941 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4eb39cc3:::benchmark_data_smithi138_477915_object11:head# 2024-09-15T05:50:31.941 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.966 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4f21a6e9:::benchmark_data_smithi138_483817_object98:head# 2024-09-15T05:50:31.966 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:31.999 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4fdb5ea1:::benchmark_data_smithi138_477915_object86:head# 2024-09-15T05:50:31.999 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.024 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4fea3b8e:::benchmark_data_smithi138_483817_object141:head# 2024-09-15T05:50:32.025 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.050 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52b28e51:::benchmark_data_smithi138_477915_object188:head# 2024-09-15T05:50:32.050 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.075 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52f1d98b:::benchmark_data_smithi138_477915_object29:head# 2024-09-15T05:50:32.075 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.083 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:535bd7b0:::benchmark_data_smithi138_483817_object21:head# 2024-09-15T05:50:32.083 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.092 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:538d02e4:::benchmark_data_smithi138_477915_object387:head# 2024-09-15T05:50:32.092 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.116 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:53b28f04:::benchmark_data_smithi138_477915_object193:head# 2024-09-15T05:50:32.117 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.142 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:543e73a1:::benchmark_data_smithi138_477915_object143:head# 2024-09-15T05:50:32.142 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.167 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:546413ea:::benchmark_data_smithi138_477915_object35:head# 2024-09-15T05:50:32.167 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.200 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54b5d110:::benchmark_data_smithi138_477915_object370:head# 2024-09-15T05:50:32.200 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.234 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54c3eb42:::benchmark_data_smithi138_483817_object171:head# 2024-09-15T05:50:32.234 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.267 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54d808a1:::benchmark_data_smithi138_477915_object147:head# 2024-09-15T05:50:32.267 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.301 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:556f515c:::benchmark_data_smithi138_477915_object367:head# 2024-09-15T05:50:32.301 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.334 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5581a3b9:::benchmark_data_smithi138_477915_object123:head# 2024-09-15T05:50:32.334 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.368 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55b4ce8c:::benchmark_data_smithi138_477915_object327:head# 2024-09-15T05:50:32.368 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.401 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55e0f391:::benchmark_data_smithi138_477915_object40:head# 2024-09-15T05:50:32.401 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.434 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:564dfa9a:::benchmark_data_smithi138_477915_object89:head# 2024-09-15T05:50:32.435 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.468 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56619472:::benchmark_data_smithi138_477915_object291:head# 2024-09-15T05:50:32.468 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.501 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56ac2e18:::benchmark_data_smithi138_477915_object373:head# 2024-09-15T05:50:32.501 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.535 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e0ff0d:::benchmark_data_smithi138_477915_object246:head# 2024-09-15T05:50:32.535 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.550 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56e54ad7:::benchmark_data_smithi138_477915_object241:head# 2024-09-15T05:50:32.550 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.559 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:570d6bf9:::benchmark_data_smithi138_483817_object3:head# 2024-09-15T05:50:32.559 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.567 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:572a26cc:::benchmark_data_smithi138_477915_object3:head# 2024-09-15T05:50:32.567 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.584 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:57365c69:::benchmark_data_smithi138_477915_object185:head# 2024-09-15T05:50:32.584 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.592 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:58b05a98:::benchmark_data_smithi138_477915_object94:head# 2024-09-15T05:50:32.592 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.601 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5979f860:::benchmark_data_smithi138_477915_object42:head# 2024-09-15T05:50:32.601 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.609 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:598335b8:::benchmark_data_smithi138_483817_object114:head# 2024-09-15T05:50:32.609 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.617 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5a3930c3:::benchmark_data_smithi138_477915_object92:head# 2024-09-15T05:50:32.617 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.626 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5acdf762:::benchmark_data_smithi138_477915_object217:head# 2024-09-15T05:50:32.626 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.651 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5b192008:::benchmark_data_smithi138_477915_object190:head# 2024-09-15T05:50:32.651 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.702 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5bd647c9:::benchmark_data_smithi138_477915_object308:head# 2024-09-15T05:50:32.702 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.777 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c5abf37:::benchmark_data_smithi138_477915_object227:head# 2024-09-15T05:50:32.777 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.828 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5cc75e94:::benchmark_data_smithi138_477915_object261:head# 2024-09-15T05:50:32.828 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.878 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5d452870:::benchmark_data_smithi138_477915_object386:head# 2024-09-15T05:50:32.878 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.928 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5e2e9211:::benchmark_data_smithi138_477915_object201:head# 2024-09-15T05:50:32.928 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:32.978 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5f10ef7e:::benchmark_data_smithi138_477915_object182:head# 2024-09-15T05:50:32.978 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.009 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fa8bc7a:::benchmark_data_smithi138_477915_object371:head# 2024-09-15T05:50:33.009 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.038 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fd3194b:::benchmark_data_smithi138_477915_object83:head# 2024-09-15T05:50:33.039 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.072 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5fec7540:::benchmark_data_smithi138_477915_object132:head# 2024-09-15T05:50:33.072 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.105 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6012bbb1:::benchmark_data_smithi138_477915_object213:head# 2024-09-15T05:50:33.105 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.139 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:620660f2:::benchmark_data_smithi138_477915_object138:head# 2024-09-15T05:50:33.139 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.172 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:62b5408f:::benchmark_data_smithi138_483817_object151:head# 2024-09-15T05:50:33.172 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.221 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:62fcea3a:::benchmark_data_smithi138_477915_object374:head# 2024-09-15T05:50:33.221 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.297 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:630b98f5:::benchmark_data_smithi138_483817_object80:head# 2024-09-15T05:50:33.297 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.347 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:635fcc02:::benchmark_data_smithi138_477915_object18:head# 2024-09-15T05:50:33.347 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.380 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:636ef2c0:::benchmark_data_smithi138_483817_object45:head# 2024-09-15T05:50:33.381 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.414 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:63be4c06:::benchmark_data_smithi138_483817_object1:head# 2024-09-15T05:50:33.414 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.447 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6429912b:::benchmark_data_smithi138_477915_object78:head# 2024-09-15T05:50:33.447 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.480 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:648cc3c2:::benchmark_data_smithi138_477915_object180:head# 2024-09-15T05:50:33.480 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.514 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64ae8f93:::benchmark_data_smithi138_477915_object273:head# 2024-09-15T05:50:33.514 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.547 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:65851d9e:::benchmark_data_smithi138_477915_object276:head# 2024-09-15T05:50:33.547 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.581 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:66acc5ee:::benchmark_data_smithi138_477915_object155:head# 2024-09-15T05:50:33.581 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.614 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:677fba78:::benchmark_data_smithi138_483817_object91:head# 2024-09-15T05:50:33.614 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.647 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:67e15da6:::benchmark_data_smithi138_477915_object253:head# 2024-09-15T05:50:33.647 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.681 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68a5ecca:::benchmark_data_smithi138_477915_object260:head# 2024-09-15T05:50:33.681 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.714 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68b33428:::benchmark_data_smithi138_477915_object388:head# 2024-09-15T05:50:33.714 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.747 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6940b9d3:::benchmark_data_smithi138_477915_object239:head# 2024-09-15T05:50:33.747 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.781 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:69f7ef4c:::benchmark_data_smithi138_477915_object127:head# 2024-09-15T05:50:33.781 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.814 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6aef21a2:::benchmark_data_smithi138_477915_object54:head# 2024-09-15T05:50:33.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.847 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6b3c2b63:::benchmark_data_smithi138_477915_object208:head# 2024-09-15T05:50:33.847 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.880 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bf4848f:::benchmark_data_smithi138_477915_object399:head# 2024-09-15T05:50:33.881 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.914 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ccf34bf:::benchmark_data_smithi138_477915_object22:head# 2024-09-15T05:50:33.914 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.947 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6cec5432:::benchmark_data_smithi138_483817_object174:head# 2024-09-15T05:50:33.947 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:33.981 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6d5a3232:::benchmark_data_smithi138_477915_object43:head# 2024-09-15T05:50:33.981 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.014 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dc7f41b:::benchmark_data_smithi138_477915_object262:head# 2024-09-15T05:50:34.014 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.047 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dd17fb9:::benchmark_data_smithi138_477915_object263:head# 2024-09-15T05:50:34.047 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.089 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6ddcb643:::benchmark_data_smithi138_477915_object285:head# 2024-09-15T05:50:34.089 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.123 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dfa6e82:::benchmark_data_smithi138_483817_object44:head# 2024-09-15T05:50:34.123 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.156 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6e1716aa:::benchmark_data_smithi138_477915_object135:head# 2024-09-15T05:50:34.156 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.189 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:702ed571:::benchmark_data_smithi138_477915_object219:head# 2024-09-15T05:50:34.189 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.222 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:707bb828:::benchmark_data_smithi138_477915_object158:head# 2024-09-15T05:50:34.222 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.256 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:70ca4348:::benchmark_data_smithi138_477915_object26:head# 2024-09-15T05:50:34.256 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.289 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7100933a:::benchmark_data_smithi138_483817_object72:head# 2024-09-15T05:50:34.289 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.322 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:710e85f3:::benchmark_data_smithi138_483817_object32:head# 2024-09-15T05:50:34.323 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.356 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:713c8310:::benchmark_data_smithi138_483817_object71:head# 2024-09-15T05:50:34.356 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.381 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71784a9a:::benchmark_data_smithi138_477915_object376:head# 2024-09-15T05:50:34.381 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.406 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:717dd36a:::benchmark_data_smithi138_477915_object139:head# 2024-09-15T05:50:34.406 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.431 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71880930:::benchmark_data_smithi138_483817_object61:head# 2024-09-15T05:50:34.431 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.456 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:71fb5440:::benchmark_data_smithi138_477915_object97:head# 2024-09-15T05:50:34.456 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.481 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7314c8d4:::benchmark_data_smithi138_477915_object152:head# 2024-09-15T05:50:34.481 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.514 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:740159d0:::benchmark_data_smithi138_477915_object0:head# 2024-09-15T05:50:34.515 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.539 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:74886e53:::benchmark_data_smithi138_477915_object112:head# 2024-09-15T05:50:34.539 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.581 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:75046150:::benchmark_data_smithi138_477915_object279:head# 2024-09-15T05:50:34.581 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.614 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:758b6ae7:::benchmark_data_smithi138_483817_object148:head# 2024-09-15T05:50:34.615 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.648 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:76871f44:::benchmark_data_smithi138_483817_object5:head# 2024-09-15T05:50:34.648 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.686 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:773e28fe:::benchmark_data_smithi138_477915_object377:head# 2024-09-15T05:50:34.686 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.736 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7743c8d9:::benchmark_data_smithi138_483817_object24:head# 2024-09-15T05:50:34.736 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.765 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7783b60c:::benchmark_data_smithi138_477915_object101:head# 2024-09-15T05:50:34.765 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.790 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:786160e4:::benchmark_data_smithi138_477915_object38:head# 2024-09-15T05:50:34.790 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.815 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:78f9c0b4:::benchmark_data_smithi138_483817_object154:head# 2024-09-15T05:50:34.815 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.840 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a2bef6b:::benchmark_data_smithi138_477915_object58:head# 2024-09-15T05:50:34.840 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.873 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7a76feec:::benchmark_data_smithi138_477915_object63:head# 2024-09-15T05:50:34.873 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.906 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7bb84b6a:::benchmark_data_smithi138_477915_object5:head# 2024-09-15T05:50:34.906 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.940 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7be49943:::benchmark_data_smithi138_477915_object48:head# 2024-09-15T05:50:34.940 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:34.973 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d1fdf77:::benchmark_data_smithi138_483817_object104:head# 2024-09-15T05:50:34.973 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.006 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d976229:::benchmark_data_smithi138_477915_object207:head# 2024-09-15T05:50:35.007 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.040 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d9d7a5f:::benchmark_data_smithi138_477915_object369:head# 2024-09-15T05:50:35.040 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.073 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7e2c4f9e:::benchmark_data_smithi138_483817_object173:head# 2024-09-15T05:50:35.073 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.123 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eaec87b:::benchmark_data_smithi138_477915_object154:head# 2024-09-15T05:50:35.123 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.157 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7eda30d0:::benchmark_data_smithi138_477915_object310:head# 2024-09-15T05:50:35.157 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.190 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7f744066:::benchmark_data_smithi138_477915_object205:head# 2024-09-15T05:50:35.190 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.255 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81770d05:::benchmark_data_smithi138_483817_object103:head# 2024-09-15T05:50:35.255 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.306 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81cead58:::benchmark_data_smithi138_477915_object95:head# 2024-09-15T05:50:35.306 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.357 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81e31d0f:::benchmark_data_smithi138_477915_object203:head# 2024-09-15T05:50:35.357 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.390 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:829c9f42:::benchmark_data_smithi138_477915_object297:head# 2024-09-15T05:50:35.390 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.423 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8326356e:::benchmark_data_smithi138_477915_object272:head# 2024-09-15T05:50:35.423 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.456 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84ba03a2:::benchmark_data_smithi138_477915_object268:head# 2024-09-15T05:50:35.457 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.490 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84bb0071:::benchmark_data_smithi138_477915_object169:head# 2024-09-15T05:50:35.490 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.523 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8531dfd7:::benchmark_data_smithi138_477915_object311:head# 2024-09-15T05:50:35.523 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.557 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:853bf743:::benchmark_data_smithi138_477915_object257:head# 2024-09-15T05:50:35.557 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.623 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:861edb1c:::benchmark_data_smithi138_483817_object118:head# 2024-09-15T05:50:35.624 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.657 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e0b007:::benchmark_data_smithi138_477915_object74:head# 2024-09-15T05:50:35.657 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.690 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:86e93436:::benchmark_data_smithi138_477915_object319:head# 2024-09-15T05:50:35.690 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.724 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8709604b:::benchmark_data_smithi138_477915_object347:head# 2024-09-15T05:50:35.724 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.757 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:87d6852e:::benchmark_data_smithi138_477915_object231:head# 2024-09-15T05:50:35.757 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.790 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:88408030:::benchmark_data_smithi138_483817_object48:head# 2024-09-15T05:50:35.790 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.824 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:88c42840:::benchmark_data_smithi138_477915_object140:head# 2024-09-15T05:50:35.824 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.857 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8919b48c:::benchmark_data_smithi138_483817_object4:head# 2024-09-15T05:50:35.857 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.890 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8974e594:::benchmark_data_smithi138_483817_object115:head# 2024-09-15T05:50:35.890 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.923 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:89a81864:::benchmark_data_smithi138_477915_object264:head# 2024-09-15T05:50:35.924 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.957 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a16f6c1:::benchmark_data_smithi138_477915_object93:head# 2024-09-15T05:50:35.957 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:35.990 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a8bf090:::benchmark_data_smithi138_483817_object138:head# 2024-09-15T05:50:35.990 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.024 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aa67a0a:::benchmark_data_smithi138_477915_object235:head# 2024-09-15T05:50:36.024 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.057 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8aedb92d:::benchmark_data_smithi138_477915_object365:head# 2024-09-15T05:50:36.057 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.090 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8b120672:::benchmark_data_smithi138_483817_object143:head# 2024-09-15T05:50:36.090 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.123 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8be95c11:::benchmark_data_smithi138_477915_object323:head# 2024-09-15T05:50:36.124 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.157 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c3675a7:::benchmark_data_smithi138_477915_object303:head# 2024-09-15T05:50:36.157 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.190 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d8d0588:::benchmark_data_smithi138_477915_object28:head# 2024-09-15T05:50:36.191 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.224 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f046a5e:::benchmark_data_smithi138_477915_object109:head# 2024-09-15T05:50:36.224 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.257 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f04eb67:::benchmark_data_smithi138_477915_object160:head# 2024-09-15T05:50:36.257 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.290 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f226de4:::benchmark_data_smithi138_483817_object167:head# 2024-09-15T05:50:36.290 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.324 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8fca75c6:::benchmark_data_smithi138_483817_object160:head# 2024-09-15T05:50:36.324 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.357 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:900c1e9a:::benchmark_data_smithi138_477915_object168:head# 2024-09-15T05:50:36.357 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.391 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9033da55:::benchmark_data_smithi138_477915_object23:head# 2024-09-15T05:50:36.391 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.424 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90dfb114:::benchmark_data_smithi138_477915_object233:head# 2024-09-15T05:50:36.424 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.457 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90f41851:::benchmark_data_smithi138_477915_object256:head# 2024-09-15T05:50:36.457 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.491 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:916fb673:::benchmark_data_smithi138_483817_object88:head# 2024-09-15T05:50:36.491 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.524 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91c6592f:::benchmark_data_smithi138_477915_object36:head# 2024-09-15T05:50:36.524 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.557 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91f02d94:::benchmark_data_smithi138_477915_object249:head# 2024-09-15T05:50:36.557 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.591 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92b0bd62:::benchmark_data_smithi138_483817_object73:head# 2024-09-15T05:50:36.591 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.624 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92efb357:::benchmark_data_smithi138_477915_object242:head# 2024-09-15T05:50:36.624 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.649 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:937f5fff:::benchmark_data_smithi138_483817_object87:head# 2024-09-15T05:50:36.649 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.686 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9432f393:::benchmark_data_smithi138_477915_object16:head# 2024-09-15T05:50:36.686 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.745 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:945330ef:::benchmark_data_smithi138_483817_object51:head# 2024-09-15T05:50:36.745 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.782 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9459b9e3:::benchmark_data_smithi138_483817_object136:head# 2024-09-15T05:50:36.783 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.816 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:95f9f07e:::benchmark_data_smithi138_483817_object139:head# 2024-09-15T05:50:36.816 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971a84a9:::benchmark_data_smithi138_477915_object149:head# 2024-09-15T05:50:36.849 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.882 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:971af863:::benchmark_data_smithi138_477915_object151:head# 2024-09-15T05:50:36.882 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.916 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:99020c09:::benchmark_data_smithi138_483817_object7:head# 2024-09-15T05:50:36.916 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.949 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a712198:::benchmark_data_smithi138_477915_object218:head# 2024-09-15T05:50:36.949 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:36.982 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9aa4d68e:::benchmark_data_smithi138_483817_object16:head# 2024-09-15T05:50:36.983 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.016 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9cba91af:::benchmark_data_smithi138_483817_object105:head# 2024-09-15T05:50:37.016 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.041 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9cbdb690:::benchmark_data_smithi138_483817_object20:head# 2024-09-15T05:50:37.041 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.066 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9d6d8ac9:::benchmark_data_smithi138_483817_object8:head# 2024-09-15T05:50:37.066 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.091 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9ddb669b:::benchmark_data_smithi138_483817_object65:head# 2024-09-15T05:50:37.091 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.116 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9df1f34f:::benchmark_data_smithi138_477915_object356:head# 2024-09-15T05:50:37.116 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.149 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9e3a6f9c:::benchmark_data_smithi138_483817_object40:head# 2024-09-15T05:50:37.149 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.183 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9e460c3c:::benchmark_data_smithi138_483817_object50:head# 2024-09-15T05:50:37.183 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.264 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9ea5731d:::benchmark_data_smithi138_483817_object79:head# 2024-09-15T05:50:37.264 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.323 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9eb64e24:::benchmark_data_smithi138_483817_object23:head# 2024-09-15T05:50:37.323 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.374 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f28ef18:::benchmark_data_smithi138_483817_object33:head# 2024-09-15T05:50:37.374 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.408 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f907d5d:::benchmark_data_smithi138_483817_object0:head# 2024-09-15T05:50:37.408 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.441 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f9adf98:::benchmark_data_smithi138_477915_object200:head# 2024-09-15T05:50:37.441 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.474 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a080aed2:::benchmark_data_smithi138_477915_object104:head# 2024-09-15T05:50:37.474 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.508 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0aeaa72:::benchmark_data_smithi138_477915_object254:head# 2024-09-15T05:50:37.508 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.541 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0d22f50:::benchmark_data_smithi138_477915_object71:head# 2024-09-15T05:50:37.541 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.574 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0fc5717:::benchmark_data_smithi138_483817_object53:head# 2024-09-15T05:50:37.575 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.608 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10a53f5:::benchmark_data_smithi138_477915_object298:head# 2024-09-15T05:50:37.608 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.641 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a10dc166:::benchmark_data_smithi138_477915_object117:head# 2024-09-15T05:50:37.642 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.674 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1460743:::benchmark_data_smithi138_477915_object334:head# 2024-09-15T05:50:37.675 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.708 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a14ca631:::benchmark_data_smithi138_477915_object105:head# 2024-09-15T05:50:37.708 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.741 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1c26988:::benchmark_data_smithi138_477915_object351:head# 2024-09-15T05:50:37.741 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.774 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a2dc0f5b:::benchmark_data_smithi138_483817_object52:head# 2024-09-15T05:50:37.775 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.808 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a2e8a8e9:::benchmark_data_smithi138_477915_object184:head# 2024-09-15T05:50:37.808 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.841 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a300e90c:::benchmark_data_smithi138_483817_object155:head# 2024-09-15T05:50:37.842 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.875 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5019f41:::benchmark_data_smithi138_477915_object302:head# 2024-09-15T05:50:37.875 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.908 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5330bbb:::benchmark_data_smithi138_483817_object149:head# 2024-09-15T05:50:37.908 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.941 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a574d19f:::benchmark_data_smithi138_477915_object27:head# 2024-09-15T05:50:37.942 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:37.974 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a58e8ebc:::benchmark_data_smithi138_483817_object39:head# 2024-09-15T05:50:37.975 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.008 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a5add657:::benchmark_data_smithi138_477915_object96:head# 2024-09-15T05:50:38.008 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.058 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a65ba2e1:::benchmark_data_smithi138_477915_object348:head# 2024-09-15T05:50:38.058 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.092 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a76a8553:::benchmark_data_smithi138_477915_object384:head# 2024-09-15T05:50:38.092 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.125 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a84031a3:::benchmark_data_smithi138_477915_object156:head# 2024-09-15T05:50:38.125 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.158 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a89e1157:::benchmark_data_smithi138_477915_object66:head# 2024-09-15T05:50:38.158 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.191 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a8ec4680:::benchmark_data_smithi138_477915_object280:head# 2024-09-15T05:50:38.191 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.225 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a91194b3:::benchmark_data_smithi138_477915_object68:head# 2024-09-15T05:50:38.225 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.258 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a9405195:::benchmark_data_smithi138_483817_object25:head# 2024-09-15T05:50:38.258 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.291 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aad81857:::benchmark_data_smithi138_483817_object146:head# 2024-09-15T05:50:38.292 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.325 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aae671fe:::benchmark_data_smithi138_477915_object173:head# 2024-09-15T05:50:38.325 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.358 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:abb9f89a:::benchmark_data_smithi138_483817_object166:head# 2024-09-15T05:50:38.358 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.392 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:abf706c3:::benchmark_data_smithi138_477915_object121:head# 2024-09-15T05:50:38.392 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.425 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ad8a20c0:::benchmark_data_smithi138_477915_object375:head# 2024-09-15T05:50:38.425 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.458 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ade77a16:::benchmark_data_smithi138_477915_object88:head# 2024-09-15T05:50:38.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.492 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ae6263f9:::benchmark_data_smithi138_477915_object31:head# 2024-09-15T05:50:38.492 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.525 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aec18c78:::benchmark_data_smithi138_483817_object116:head# 2024-09-15T05:50:38.525 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.559 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:af390588:::benchmark_data_smithi138_483817_object41:head# 2024-09-15T05:50:38.559 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.591 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b02634ce:::benchmark_data_smithi138_483817_object108:head# 2024-09-15T05:50:38.592 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.625 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b05f7e87:::benchmark_data_smithi138_477915_object316:head# 2024-09-15T05:50:38.625 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.658 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b0627822:::benchmark_data_smithi138_477915_object255:head# 2024-09-15T05:50:38.658 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.711 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b08f0d28:::benchmark_data_smithi138_477915_object229:head# 2024-09-15T05:50:38.711 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.758 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b1344ffa:::benchmark_data_smithi138_477915_object33:head# 2024-09-15T05:50:38.759 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.792 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b142aea1:::benchmark_data_smithi138_477915_object64:head# 2024-09-15T05:50:38.792 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.825 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b44163:::benchmark_data_smithi138_483817_object74:head# 2024-09-15T05:50:38.825 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.875 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b53955:::benchmark_data_smithi138_477915_object199:head# 2024-09-15T05:50:38.875 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.900 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b2b94858:::benchmark_data_smithi138_477915_object345:head# 2024-09-15T05:50:38.900 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.925 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3064d98:::benchmark_data_smithi138_483817_object26:head# 2024-09-15T05:50:38.925 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.950 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b30ed2d9:::benchmark_data_smithi138_477915_object224:head# 2024-09-15T05:50:38.950 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.967 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b36e2fb6:::benchmark_data_smithi138_477915_object56:head# 2024-09-15T05:50:38.967 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:38.992 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b388f84c:::benchmark_data_smithi138_477915_object270:head# 2024-09-15T05:50:38.992 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.017 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3bb6a6a:::benchmark_data_smithi138_477915_object251:head# 2024-09-15T05:50:39.017 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.042 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b3ceb653:::benchmark_data_smithi138_477915_object130:head# 2024-09-15T05:50:39.042 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.075 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b413d550:::benchmark_data_smithi138_477915_object76:head# 2024-09-15T05:50:39.075 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.109 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b49d13a3:::benchmark_data_smithi138_477915_object296:head# 2024-09-15T05:50:39.109 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.142 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b4dcf7e3:::benchmark_data_smithi138_477915_object161:head# 2024-09-15T05:50:39.142 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.175 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b51b59a9:::benchmark_data_smithi138_477915_object159:head# 2024-09-15T05:50:39.176 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.222 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b52536fd:::benchmark_data_smithi138_477915_object102:head# 2024-09-15T05:50:39.222 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.306 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b711d581:::benchmark_data_smithi138_477915_object167:head# 2024-09-15T05:50:39.306 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.359 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7914ecd:::benchmark_data_smithi138_477915_object343:head# 2024-09-15T05:50:39.359 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.392 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b7ce732e:::benchmark_data_smithi138_477915_object357:head# 2024-09-15T05:50:39.392 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.426 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8000b14:::benchmark_data_smithi138_477915_object103:head# 2024-09-15T05:50:39.426 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:50:39.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.526 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b89297e9:::benchmark_data_smithi138_477915_object362:head# 2024-09-15T05:50:39.526 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.561 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b895e95c:::benchmark_data_smithi138_477915_object359:head# 2024-09-15T05:50:39.561 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.592 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b897bd1c:::benchmark_data_smithi138_477915_object226:head# 2024-09-15T05:50:39.592 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.626 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8c52179:::benchmark_data_smithi138_483817_object125:head# 2024-09-15T05:50:39.626 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.659 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8f15cb8:::benchmark_data_smithi138_477915_object136:head# 2024-09-15T05:50:39.659 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.692 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bacecf10:::benchmark_data_smithi138_477915_object340:head# 2024-09-15T05:50:39.692 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.726 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bcda79f2:::benchmark_data_smithi138_477915_object228:head# 2024-09-15T05:50:39.726 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.761 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bdbfac57:::benchmark_data_smithi138_477915_object171:head# 2024-09-15T05:50:39.761 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.792 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bde3dd73:::benchmark_data_smithi138_483817_object145:head# 2024-09-15T05:50:39.792 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.826 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:be946f7d:::benchmark_data_smithi138_477915_object390:head# 2024-09-15T05:50:39.826 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.859 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bf91c211:::benchmark_data_smithi138_477915_object162:head# 2024-09-15T05:50:39.859 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.892 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bf9471cf:::benchmark_data_smithi138_483817_object144:head# 2024-09-15T05:50:39.892 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.926 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c080fdda:::benchmark_data_smithi138_477915_object354:head# 2024-09-15T05:50:39.926 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:39.959 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c0be2264:::benchmark_data_smithi138_483817_object120:head# 2024-09-15T05:50:39.959 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.001 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c0cda59f:::benchmark_data_smithi138_483817_object57:head# 2024-09-15T05:50:40.001 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.034 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c11f897f:::benchmark_data_smithi138_477915_object236:head# 2024-09-15T05:50:40.035 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.067 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c122f290:::benchmark_data_smithi138_477915_object172:head# 2024-09-15T05:50:40.068 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.101 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c1ed322f:::benchmark_data_smithi138_477915_object129:head# 2024-09-15T05:50:40.101 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.134 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2727a0a:::benchmark_data_smithi138_483817_object163:head# 2024-09-15T05:50:40.135 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.168 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2bef58a:::benchmark_data_smithi138_477915_object238:head# 2024-09-15T05:50:40.169 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.201 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c2e2e8b4:::benchmark_data_smithi138_477915_object234:head# 2024-09-15T05:50:40.201 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.235 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c36b3bbe:::benchmark_data_smithi138_477915_object247:head# 2024-09-15T05:50:40.235 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.268 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c38e75c6:::benchmark_data_smithi138_483817_object119:head# 2024-09-15T05:50:40.268 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.301 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c46189c0:::benchmark_data_smithi138_477915_object300:head# 2024-09-15T05:50:40.301 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.335 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4b7580a:::benchmark_data_smithi138_483817_object100:head# 2024-09-15T05:50:40.335 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.368 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4c0edd9:::benchmark_data_smithi138_477915_object87:head# 2024-09-15T05:50:40.369 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.402 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4e6b38c:::benchmark_data_smithi138_483817_object110:head# 2024-09-15T05:50:40.402 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.435 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5229070:::benchmark_data_smithi138_477915_object237:head# 2024-09-15T05:50:40.435 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.468 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5a6edda:::benchmark_data_smithi138_483817_object112:head# 2024-09-15T05:50:40.469 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.502 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5aafc40:::benchmark_data_smithi138_477915_object21:head# 2024-09-15T05:50:40.502 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.535 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5bcceac:::benchmark_data_smithi138_477915_object314:head# 2024-09-15T05:50:40.535 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.568 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5cf901b:::benchmark_data_smithi138_477915_object69:head# 2024-09-15T05:50:40.568 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.601 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c612e5a0:::benchmark_data_smithi138_477915_object131:head# 2024-09-15T05:50:40.602 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.635 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c686c228:::benchmark_data_smithi138_477915_object383:head# 2024-09-15T05:50:40.635 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.668 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c6d1efb1:::benchmark_data_smithi138_477915_object269:head# 2024-09-15T05:50:40.668 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.745 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7b09ada:::benchmark_data_smithi138_477915_object350:head# 2024-09-15T05:50:40.745 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.776 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7e9582c:::benchmark_data_smithi138_477915_object318:head# 2024-09-15T05:50:40.776 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.810 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8129485:::benchmark_data_smithi138_477915_object400:head# 2024-09-15T05:50:40.810 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.843 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c81375d4:::benchmark_data_smithi138_477915_object186:head# 2024-09-15T05:50:40.843 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.876 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c81e2b79:::benchmark_data_smithi138_483817_object95:head# 2024-09-15T05:50:40.876 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.910 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c836a7f2:::benchmark_data_smithi138_477915_object392:head# 2024-09-15T05:50:40.910 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.952 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c84b5188:::benchmark_data_smithi138_477915_object282:head# 2024-09-15T05:50:40.952 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:40.985 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8ed4655:::benchmark_data_smithi138_483817_object28:head# 2024-09-15T05:50:40.985 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.018 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8f99db7:::benchmark_data_smithi138_477915_object295:head# 2024-09-15T05:50:41.018 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.052 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c918d639:::benchmark_data_smithi138_483817_object55:head# 2024-09-15T05:50:41.052 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.085 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c94caa86:::benchmark_data_smithi138_483817_object140:head# 2024-09-15T05:50:41.085 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.118 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c957d3d7:::benchmark_data_smithi138_483817_object59:head# 2024-09-15T05:50:41.119 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.152 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ca8c3be3:::benchmark_data_smithi138_483817_object134:head# 2024-09-15T05:50:41.152 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.185 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cac2fc5a:::benchmark_data_smithi138_477915_object380:head# 2024-09-15T05:50:41.185 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.264 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cb485fde:::benchmark_data_smithi138_477915_object342:head# 2024-09-15T05:50:41.264 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.306 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cb97c1ba:::benchmark_data_smithi138_483817_object165:head# 2024-09-15T05:50:41.306 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.360 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cbb3fcfe:::benchmark_data_smithi138_477915_object258:head# 2024-09-15T05:50:41.360 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.393 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd46d84a:::benchmark_data_smithi138_477915_object337:head# 2024-09-15T05:50:41.393 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.427 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd70a88a:::benchmark_data_smithi138_483817_object161:head# 2024-09-15T05:50:41.427 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.460 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd852d6c:::benchmark_data_smithi138_477915_object107:head# 2024-09-15T05:50:41.460 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.485 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd8a05cc:::benchmark_data_smithi138_477915_object2:head# 2024-09-15T05:50:41.485 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.510 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ce5af95b:::benchmark_data_smithi138_483817_object147:head# 2024-09-15T05:50:41.510 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.535 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ce8b7145:::benchmark_data_smithi138_483817_object78:head# 2024-09-15T05:50:41.535 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.560 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cec5b2bb:::benchmark_data_smithi138_477915_object25:head# 2024-09-15T05:50:41.560 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.585 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d0e38665:::benchmark_data_smithi138_477915_object266:head# 2024-09-15T05:50:41.585 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.610 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d165ae0f:::benchmark_data_smithi138_477915_object211:head# 2024-09-15T05:50:41.610 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.635 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d20fbe61:::benchmark_data_smithi138_477915_object284:head# 2024-09-15T05:50:41.635 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.669 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d2de79dc:::benchmark_data_smithi138_483817_object94:head# 2024-09-15T05:50:41.669 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.702 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d32e8031:::benchmark_data_smithi138_483817_object81:head# 2024-09-15T05:50:41.702 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.735 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d350cfaa:::benchmark_data_smithi138_477915_object113:head# 2024-09-15T05:50:41.735 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.769 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d3967d82:::benchmark_data_smithi138_483817_object58:head# 2024-09-15T05:50:41.769 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.802 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d3b5e940:::benchmark_data_smithi138_477915_object14:head# 2024-09-15T05:50:41.802 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.835 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d3b64364:::benchmark_data_smithi138_483817_object36:head# 2024-09-15T05:50:41.835 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.869 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d41bfbfa:::benchmark_data_smithi138_483817_object12:head# 2024-09-15T05:50:41.869 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.902 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d48aafda:::benchmark_data_smithi138_477915_object358:head# 2024-09-15T05:50:41.902 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.935 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d567f524:::benchmark_data_smithi138_483817_object169:head# 2024-09-15T05:50:41.935 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:41.969 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5a212a5:::benchmark_data_smithi138_477915_object106:head# 2024-09-15T05:50:41.969 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.002 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5d6c19c:::benchmark_data_smithi138_477915_object355:head# 2024-09-15T05:50:42.002 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.035 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6691603:::benchmark_data_smithi138_483817_object22:head# 2024-09-15T05:50:42.036 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.069 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6d89b5a:::benchmark_data_smithi138_477915_object341:head# 2024-09-15T05:50:42.069 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.102 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d789c5f7:::benchmark_data_smithi138_477915_object336:head# 2024-09-15T05:50:42.102 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.160 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c4ab31:::benchmark_data_smithi138_477915_object189:head# 2024-09-15T05:50:42.160 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.235 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7c508f3:::benchmark_data_smithi138_477915_object4:head# 2024-09-15T05:50:42.235 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.319 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7ebcc2e:::benchmark_data_smithi138_477915_object265:head# 2024-09-15T05:50:42.319 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.419 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d878fde1:::benchmark_data_smithi138_483817_object93:head# 2024-09-15T05:50:42.419 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.495 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8daf539:::benchmark_data_smithi138_477915_object99:head# 2024-09-15T05:50:42.495 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.527 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8e85bad:::benchmark_data_smithi138_477915_object166:head# 2024-09-15T05:50:42.527 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.561 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d95dedd9:::benchmark_data_smithi138_477915_object274:head# 2024-09-15T05:50:42.561 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.594 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da010958:::benchmark_data_smithi138_483817_object85:head# 2024-09-15T05:50:42.594 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.644 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da2f71a7:::benchmark_data_smithi138_477915_object65:head# 2024-09-15T05:50:42.644 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.695 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da407726:::benchmark_data_smithi138_477915_object179:head# 2024-09-15T05:50:42.695 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.762 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da7cec77:::benchmark_data_smithi138_483817_object102:head# 2024-09-15T05:50:42.763 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.794 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dabfd9f7:::benchmark_data_smithi138_477915_object118:head# 2024-09-15T05:50:42.794 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.827 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dad5a76a:::benchmark_data_smithi138_477915_object305:head# 2024-09-15T05:50:42.827 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.861 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd1177ff:::benchmark_data_smithi138_483817_object47:head# 2024-09-15T05:50:42.861 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.894 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6aea31:::benchmark_data_smithi138_477915_object150:head# 2024-09-15T05:50:42.894 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.928 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd78a891:::benchmark_data_smithi138_477915_object353:head# 2024-09-15T05:50:42.928 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.961 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:df8c54da:::benchmark_data_smithi138_483817_object111:head# 2024-09-15T05:50:42.961 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:42.994 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfa1d077:::benchmark_data_smithi138_483817_object17:head# 2024-09-15T05:50:42.994 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.028 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfac43cc:::benchmark_data_smithi138_477915_object126:head# 2024-09-15T05:50:43.028 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.061 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dfe8aa68:::benchmark_data_smithi138_477915_object51:head# 2024-09-15T05:50:43.061 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.103 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e0895a86:::benchmark_data_smithi138_483817_object158:head# 2024-09-15T05:50:43.103 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.136 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e100c584:::benchmark_data_smithi138_477915_object8:head# 2024-09-15T05:50:43.136 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.169 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1828df5:::benchmark_data_smithi138_477915_object267:head# 2024-09-15T05:50:43.169 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.231 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1ad5afa:::benchmark_data_smithi138_483817_object30:head# 2024-09-15T05:50:43.231 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.307 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e1daa4ae:::benchmark_data_smithi138_477915_object37:head# 2024-09-15T05:50:43.307 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.353 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e23cd4da:::benchmark_data_smithi138_477915_object32:head# 2024-09-15T05:50:43.353 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e436bdb7:::benchmark_data_smithi138_483817_object68:head# 2024-09-15T05:50:43.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.444 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4da1a27:::benchmark_data_smithi138_477915_object15:head# 2024-09-15T05:50:43.444 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.478 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4f1a68f:::benchmark_data_smithi138_477915_object395:head# 2024-09-15T05:50:43.478 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.511 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e575c0b5:::benchmark_data_smithi138_483817_object157:head# 2024-09-15T05:50:43.511 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.544 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e6fdeca3:::benchmark_data_smithi138_477915_object338:head# 2024-09-15T05:50:43.545 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.578 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7c862ab:::benchmark_data_smithi138_477915_object119:head# 2024-09-15T05:50:43.578 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.611 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e7e5da90:::benchmark_data_smithi138_477915_object393:head# 2024-09-15T05:50:43.611 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.636 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e922bb7a:::benchmark_data_smithi138_483817_object162:head# 2024-09-15T05:50:43.636 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.661 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e9e1dfd6:::benchmark_data_smithi138_477915_object315:head# 2024-09-15T05:50:43.661 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.686 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eaee7ade:::benchmark_data_smithi138_477915_object277:head# 2024-09-15T05:50:43.686 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.720 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb2d0c81:::benchmark_data_smithi138_477915_object352:head# 2024-09-15T05:50:43.720 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.753 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eb74e80f:::benchmark_data_smithi138_477915_object39:head# 2024-09-15T05:50:43.753 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.786 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ebaa8086:::benchmark_data_smithi138_483817_object123:head# 2024-09-15T05:50:43.786 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.820 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ebd302e1:::benchmark_data_smithi138_477915_object221:head# 2024-09-15T05:50:43.820 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.853 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ebde0c8a:::benchmark_data_smithi138_483817_object170:head# 2024-09-15T05:50:43.853 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.886 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec58d5d5:::benchmark_data_smithi138_483817_object97:head# 2024-09-15T05:50:43.887 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:43.970 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec96583a:::benchmark_data_smithi138_477915_object6:head# 2024-09-15T05:50:43.970 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.003 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecd13f7c:::benchmark_data_smithi138_477915_object174:head# 2024-09-15T05:50:44.003 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.037 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecda3af5:::benchmark_data_smithi138_477915_object7:head# 2024-09-15T05:50:44.038 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.070 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed2124e8:::benchmark_data_smithi138_477915_object283:head# 2024-09-15T05:50:44.070 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.103 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed4f8904:::benchmark_data_smithi138_477915_object321:head# 2024-09-15T05:50:44.103 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.136 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eec767c6:::benchmark_data_smithi138_477915_object304:head# 2024-09-15T05:50:44.137 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.170 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eedebaa9:::benchmark_data_smithi138_483817_object121:head# 2024-09-15T05:50:44.170 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.203 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef20ff5b:::benchmark_data_smithi138_477915_object148:head# 2024-09-15T05:50:44.203 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.236 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef304116:::benchmark_data_smithi138_477915_object49:head# 2024-09-15T05:50:44.237 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.270 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef4d9672:::benchmark_data_smithi138_483817_object126:head# 2024-09-15T05:50:44.270 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.303 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef573daf:::benchmark_data_smithi138_477915_object177:head# 2024-09-15T05:50:44.303 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.336 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:effa6446:::benchmark_data_smithi138_483817_object150:head# 2024-09-15T05:50:44.337 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.370 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f0690f60:::benchmark_data_smithi138_477915_object397:head# 2024-09-15T05:50:44.370 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.403 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f15d302a:::benchmark_data_smithi138_477915_object81:head# 2024-09-15T05:50:44.403 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.437 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f2a7b9b6:::benchmark_data_smithi138_483817_object164:head# 2024-09-15T05:50:44.437 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.470 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f2b2bc3f:::benchmark_data_smithi138_477915_object216:head# 2024-09-15T05:50:44.470 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.504 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f314e879:::benchmark_data_smithi138_477915_object290:head# 2024-09-15T05:50:44.504 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.537 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f358deca:::benchmark_data_smithi138_477915_object278:head# 2024-09-15T05:50:44.537 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.570 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f35a9e64:::benchmark_data_smithi138_483817_object84:head# 2024-09-15T05:50:44.570 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.603 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f3ae3ae7:::benchmark_data_smithi138_483817_object14:head# 2024-09-15T05:50:44.603 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.637 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f40ce7f6:::benchmark_data_smithi138_477915_object9:head# 2024-09-15T05:50:44.637 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.670 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f48dff42:::benchmark_data_smithi138_477915_object372:head# 2024-09-15T05:50:44.670 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.754 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4a59f6a:::benchmark_data_smithi138_477915_object12:head# 2024-09-15T05:50:44.754 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4d52c7d:::benchmark_data_smithi138_477915_object183:head# 2024-09-15T05:50:44.787 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.820 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4ffa0f0:::benchmark_data_smithi138_483817_object27:head# 2024-09-15T05:50:44.821 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.853 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f50e2ea1:::benchmark_data_smithi138_477915_object194:head# 2024-09-15T05:50:44.854 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.887 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5165677:::benchmark_data_smithi138_477915_object346:head# 2024-09-15T05:50:44.887 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.920 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5ba49b3:::benchmark_data_smithi138_477915_object333:head# 2024-09-15T05:50:44.921 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.954 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5c42dd1:::benchmark_data_smithi138_477915_object85:head# 2024-09-15T05:50:44.954 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:44.987 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f64f6100:::benchmark_data_smithi138_477915_object163:head# 2024-09-15T05:50:44.987 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.020 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f664855f:::benchmark_data_smithi138_483817_object90:head# 2024-09-15T05:50:45.021 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.053 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f6daa659:::benchmark_data_smithi138_477915_object250:head# 2024-09-15T05:50:45.053 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.095 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f7f49d03:::benchmark_data_smithi138_477915_object324:head# 2024-09-15T05:50:45.095 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.128 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f956b667:::benchmark_data_smithi138_483817_object89:head# 2024-09-15T05:50:45.129 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.162 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f9630340:::benchmark_data_smithi138_483817_object46:head# 2024-09-15T05:50:45.162 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.214 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f964d056:::benchmark_data_smithi138_483817_object159:head# 2024-09-15T05:50:45.214 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.281 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa26fae3:::benchmark_data_smithi138_477915_object73:head# 2024-09-15T05:50:45.281 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.340 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa6cc2ae:::benchmark_data_smithi138_477915_object91:head# 2024-09-15T05:50:45.340 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.387 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa9ab79f:::benchmark_data_smithi138_477915_object72:head# 2024-09-15T05:50:45.387 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.420 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fb6a8c6a:::benchmark_data_smithi138_477915_object122:head# 2024-09-15T05:50:45.420 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.454 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fba968a6:::benchmark_data_smithi138_477915_object368:head# 2024-09-15T05:50:45.454 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.487 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fd402ec6:::benchmark_data_smithi138_477915_object67:head# 2024-09-15T05:50:45.487 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.520 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fdef6d22:::benchmark_data_smithi138_477915_object309:head# 2024-09-15T05:50:45.520 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.554 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fe273b33:::benchmark_data_smithi138_477915_object146:head# 2024-09-15T05:50:45.554 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.579 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fea7c25b:::benchmark_data_smithi138_477915_object344:head# 2024-09-15T05:50:45.579 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.620 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ff1959bb:::benchmark_data_smithi138_477915_object80:head# 2024-09-15T05:50:45.621 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.654 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffb3fc0b:::benchmark_data_smithi138_477915_object115:head# 2024-09-15T05:50:45.654 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.687 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffbe43c0:::benchmark_data_smithi138_477915_object125:head# 2024-09-15T05:50:45.687 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:50:45.720 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 224 info 1.0( v 224'387 (9'209,224'387] local-lis/les=23/24 n=580 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-09-15T05:50:45.754 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:50:46.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:50:46.276 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:50:46.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:50:46.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:50:46.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:50:46.278 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:50:46.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:50:46.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:46.300+0000 7fc577e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:46.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:46.303+0000 7fc577e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:46.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:46.304+0000 7fc577e4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:50:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:50:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:50:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:50:46.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:50:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:50:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:50:46.665 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:50:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:50:46.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:50:46.822 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:46.821+0000 7fc577e4f440 -1 Falling back to public interface 2024-09-15T05:50:47.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:50:48.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:50:48.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:50:48.024 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:50:48.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:50:48.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:50:48.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:50:48.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:50:49.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:50:49.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:50:49.382 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:50:49.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:50:49.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:50:49.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:50:49.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:50:50.220 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:50.219+0000 7fc577e4f440 -1 osd.0 228 log_to_monitors true 2024-09-15T05:50:50.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:50:50.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:50:50.788 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:50:50.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:50:50.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:50:50.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:50:51.231 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 231 up_thru 231 down_at 229 last_clean_interval [23,228) [v2:127.0.0.1:6802/487621,v1:127.0.0.1:6803/487621] [v2:127.0.0.1:6804/487621,v1:127.0.0.1:6805/487621] exists,up be0cdffb-84ca-4a98-9332-b521f602a977 2024-09-15T05:50:51.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:50:51.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:50:51.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:50:51.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:50:51.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:50:51.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:50:51.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:50:51.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:50:51.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:50:51.319 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:50:51.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:50:51.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:50:51.665 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:50:51.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=992137445379 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 992137445379 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-992137445379' 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-992137445379 2024-09-15T05:50:51.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:50:51.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:50:51.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-992137445379 2024-09-15T05:50:51.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:50:51.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=992137445379 2024-09-15T05:50:51.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 992137445379' 2024-09-15T05:50:51.776 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 992137445379 2024-09-15T05:50:51.776 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:50:52.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 992137445379 2024-09-15T05:50:52.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:50:53.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:50:53.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:50:53.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 992137445379 -lt 992137445379 2024-09-15T05:50:53.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:50:53.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:50:53.484 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:50:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:50:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:50:53.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:50:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:50:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:50:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:50:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:50:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:50:54.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:50:54.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:50:54.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:50:54.334 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:50:54.810 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:50:54.811 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:50:54.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:50:54.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:50:54.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:50:54.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:50:54.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:50:54.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:50:54.917 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:50:54.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:50:54.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:50:54.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:54.937 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:54.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-09-15T05:50:54.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:50:54.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:50:54.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:50:54.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:50:54.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:50:54.941 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:50:54.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:50:54.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:50:54.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:50:54.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:50:54.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:50:54.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:50:54.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:50:54.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:50:54.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:50:54.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:50:54.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:50:54.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:54.947 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:54.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:50:54.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:50:54.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:50:54.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-09-15T05:50:54.949 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:50:54.949 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:54.949 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:54.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.466970 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:50:54.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-09-15T05:50:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:50:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:50:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:50:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-09-15T05:50:54.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:55.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-15T05:50:55.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:50:55.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:50:55.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:50:55.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:50:55.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get fsid 2024-09-15T05:50:55.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:50:55.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:50:55.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:50:55.551 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.466970/ceph-mon.a.asok 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:50:55.552 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.466970/ceph-mon.a.asok config get mon_host 2024-09-15T05:50:55.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-09-15T05:50:55.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-09-15T05:50:55.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:50:55.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:50:55.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:50:55.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-09-15T05:50:55.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:50:56.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:56.449 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:50:56.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:50:56.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-09-15T05:50:56.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-09-15T05:50:56.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:50:56.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:50:56.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:50:56.479 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:50:56.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:50:56.480 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:50:56.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 32ad5dc5-562e-406a-beed-5a5ad5ffb272' 2024-09-15T05:50:56.481 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:50:56.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDAdeZmaalVHRAAWdunkSIEKnIaLbBJICnudQ== 2024-09-15T05:50:56.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDAdeZmaalVHRAAWdunkSIEKnIaLbBJICnudQ=="}' 2024-09-15T05:50:56.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 32ad5dc5-562e-406a-beed-5a5ad5ffb272 -i td/pg-split-merge/0/new.json 2024-09-15T05:50:56.764 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:50:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-09-15T05:50:56.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDAdeZmaalVHRAAWdunkSIEKnIaLbBJICnudQ== --osd-uuid 32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:50:56.800 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.800+0000 7f60103fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:56.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.810+0000 7f60103fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:56.812 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.811+0000 7f60103fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:50:56.812 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.812+0000 7f60103fe440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:50:56.813 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.812+0000 7f60103fe440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:50:56.813 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.812+0000 7f60103fe440 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-09-15T05:50:56.851 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:50:56.850+0000 7f60103fe440 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-09-15T05:51:02.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-09-15T05:51:02.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:51:02.103 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:51:02.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:51:02.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:51:02.581 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:51:02.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:51:02.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:51:02.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:51:02.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:51:02.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:51:02.606 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:02.605+0000 7fdc5ce89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:02.607 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:02.607+0000 7fdc5ce89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:02.608 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:02.608+0000 7fdc5ce89440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:51:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:51:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:51:02.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:02.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:03.123 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:03.122+0000 7fdc5ce89440 -1 Falling back to public interface 2024-09-15T05:51:03.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:04.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:51:04.286 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:51:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:04.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:04.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:05.495 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:05.494+0000 7fdc5ce89440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:51:05.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:05.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:05.647 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:51:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:51:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:05.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:06.054 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:07.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:07.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:07.056 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:51:07.056 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:51:07.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:07.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:07.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:08.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:08.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:08.516 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:51:08.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:51:08.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:08.517 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:08.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:09.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:09.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:09.899 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:51:09.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:51:09.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:09.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:10.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:11.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:11.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:11.286 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:51:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:51:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:11.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:11.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:12.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:12.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:12.643 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:51:12.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:51:12.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:12.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:13.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:14.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:14.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:14.153 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:51:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:51:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:14.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:14.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:15.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:15.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:15.548 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:51:15.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:51:15.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:15.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:15.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:16.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:16.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:16.909 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:51:16.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:51:16.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:16.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:17.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:18.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:18.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:18.726 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:51:18.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:51:18.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:18.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:19.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:20.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:20.277 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:51:20.277 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:20.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:51:20.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:20.278 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:20.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:21.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:21.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:21.788 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:51:21.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:51:21.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:21.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:22.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:23.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:23.340 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:51:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:51:23.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:23.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:24.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:25.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:25.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:25.048 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:51:25.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:51:25.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:25.048 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:25.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:26.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:26.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:26.787 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:51:26.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:51:26.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:26.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:27.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:28.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:28.674 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:51:28.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:28.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:51:28.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:28.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:29.032 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/489519,v1:127.0.0.1:6803/489519] [v2:127.0.0.1:6804/489519,v1:127.0.0.1:6805/489519] exists,up 32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:51:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:51:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:51:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:51:29.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-09-15T05:51:29.551 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'foo' already exists 2024-09-15T05:51:29.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-09-15T05:51:29.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:51:29.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:51:29.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:51:29.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:51:29.565 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:51:29.565 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:51:29.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:51:29.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:51:29.566 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:51:29.649 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:51:30.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:51:30.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:51:30.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:51:30.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:51:30.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836484 2024-09-15T05:51:30.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836484 2024-09-15T05:51:30.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836484' 2024-09-15T05:51:30.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:51:30.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836484 2024-09-15T05:51:30.240 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:51:30.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:51:30.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836484 2024-09-15T05:51:30.241 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:51:30.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836484 2024-09-15T05:51:30.242 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-15T05:51:30.242 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836484 2024-09-15T05:51:30.242 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:51:30.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-15T05:51:30.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:51:31.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:51:31.678 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:51:32.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836484 -lt 21474836484 2024-09-15T05:51:32.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:51:32.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:51:32.044 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:51:32.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:51:32.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:51:32.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:51:32.515 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:51:32.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:51:32.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:51:32.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:51:32.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:51:32.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:51:32.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:51:32.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:51:32.856 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:51:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:51:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:51:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:51:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-09-15T05:51:33.348 INFO:tasks.workunit.client.0.smithi138.stdout:hints = 1 2024-09-15T05:51:33.349 INFO:tasks.workunit.client.0.smithi138.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-09-15T05:51:33.349 INFO:tasks.workunit.client.0.smithi138.stdout:Object prefix: benchmark_data_smithi138_490684 2024-09-15T05:51:33.349 INFO:tasks.workunit.client.0.smithi138.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-15T05:51:33.349 INFO:tasks.workunit.client.0.smithi138.stdout: 0 0 0 0 0 0 - 0 2024-09-15T05:51:34.349 INFO:tasks.workunit.client.0.smithi138.stdout: 1 16 128 112 0.109358 0.109375 0.100068 0.13652 2024-09-15T05:51:35.349 INFO:tasks.workunit.client.0.smithi138.stdout: 2 16 273 257 0.125471 0.141602 0.0754264 0.120931 2024-09-15T05:51:36.349 INFO:tasks.workunit.client.0.smithi138.stdout: 3 16 448 432 0.140607 0.170898 0.100503 0.11083 2024-09-15T05:51:37.349 INFO:tasks.workunit.client.0.smithi138.stdout:Total time run: 3.09667 2024-09-15T05:51:37.349 INFO:tasks.workunit.client.0.smithi138.stdout:Total writes made: 448 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Write size: 1024 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Object size: 1024 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Bandwidth (MB/sec): 0.141281 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Bandwidth: 0.0307733 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Max bandwidth (MB/sec): 0.170898 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Min bandwidth (MB/sec): 0.109375 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Average IOPS: 144 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev IOPS: 31.5119 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Max IOPS: 175 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Min IOPS: 112 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Average Latency(s): 0.110466 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Stddev Latency(s): 0.0300832 2024-09-15T05:51:37.350 INFO:tasks.workunit.client.0.smithi138.stdout:Max latency(s): 0.18473 2024-09-15T05:51:37.351 INFO:tasks.workunit.client.0.smithi138.stdout:Min latency(s): 0.0670492 2024-09-15T05:51:37.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-15T05:51:37.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:51:37.407 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:51:37.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:51:37.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:51:37.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:51:37.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:51:37.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-09-15T05:51:39.034 INFO:tasks.workunit.client.0.smithi138.stderr:Exporting 1.0 info 1.0( v 9'449 (0'0,9'449] local-lis/les=7/8 n=449 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:51:39.037 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:004cdd36:::benchmark_data_smithi138_490684_object63:head# 2024-09-15T05:51:39.038 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:025d89ed:::benchmark_data_smithi138_490684_object59:head# 2024-09-15T05:51:39.038 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:035a4cdf:::benchmark_data_smithi138_490684_object47:head# 2024-09-15T05:51:39.039 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:03afc33d:::benchmark_data_smithi138_490684_object319:head# 2024-09-15T05:51:39.040 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:03df6c74:::benchmark_data_smithi138_490684_object159:head# 2024-09-15T05:51:39.040 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:05e264b4:::benchmark_data_smithi138_490684_object445:head# 2024-09-15T05:51:39.041 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:060f17a0:::benchmark_data_smithi138_490684_object98:head# 2024-09-15T05:51:39.041 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:086e9542:::benchmark_data_smithi138_490684_object4:head# 2024-09-15T05:51:39.042 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0896a6d4:::benchmark_data_smithi138_490684_object166:head# 2024-09-15T05:51:39.043 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:08b73443:::benchmark_data_smithi138_490684_object148:head# 2024-09-15T05:51:39.043 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:09452e5b:::benchmark_data_smithi138_490684_object7:head# 2024-09-15T05:51:39.044 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0982f4db:::benchmark_data_smithi138_490684_object357:head# 2024-09-15T05:51:39.044 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a33106e:::benchmark_data_smithi138_490684_object179:head# 2024-09-15T05:51:39.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0a97a929:::benchmark_data_smithi138_490684_object84:head# 2024-09-15T05:51:39.045 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ac695c2:::benchmark_data_smithi138_490684_object384:head# 2024-09-15T05:51:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0ad5cfb3:::benchmark_data_smithi138_490684_object105:head# 2024-09-15T05:51:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0b6efcad:::benchmark_data_smithi138_490684_object95:head# 2024-09-15T05:51:39.046 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0c0d8acc:::benchmark_data_smithi138_490684_object21:head# 2024-09-15T05:51:39.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0c4fc3fa:::benchmark_data_smithi138_490684_object407:head# 2024-09-15T05:51:39.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0c5d3ae4:::benchmark_data_smithi138_490684_object399:head# 2024-09-15T05:51:39.047 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0d22a523:::benchmark_data_smithi138_490684_object243:head# 2024-09-15T05:51:39.048 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0d766a99:::benchmark_data_smithi138_490684_object55:head# 2024-09-15T05:51:39.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:0fcf5f0b:::benchmark_data_smithi138_490684_object237:head# 2024-09-15T05:51:39.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1079a170:::benchmark_data_smithi138_490684_object113:head# 2024-09-15T05:51:39.049 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10d8b0e9:::benchmark_data_smithi138_490684_object340:head# 2024-09-15T05:51:39.051 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10dda1b3:::benchmark_data_smithi138_490684_object236:head# 2024-09-15T05:51:39.051 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:10eb10a7:::benchmark_data_smithi138_490684_object104:head# 2024-09-15T05:51:39.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:118cebac:::benchmark_data_smithi138_490684_object189:head# 2024-09-15T05:51:39.052 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:12914623:::benchmark_data_smithi138_490684_object91:head# 2024-09-15T05:51:39.053 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:12e09e1e:::benchmark_data_smithi138_490684_object197:head# 2024-09-15T05:51:39.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:138b861d:::benchmark_data_smithi138_490684_object333:head# 2024-09-15T05:51:39.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:14292895:::benchmark_data_smithi138_490684_object266:head# 2024-09-15T05:51:39.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:14a6f7c5:::benchmark_data_smithi138_490684_object10:head# 2024-09-15T05:51:39.054 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:14cb2316:::benchmark_data_smithi138_490684_object58:head# 2024-09-15T05:51:39.055 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:158dca27:::benchmark_data_smithi138_490684_object56:head# 2024-09-15T05:51:39.055 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15907fb0:::benchmark_data_smithi138_490684_object156:head# 2024-09-15T05:51:39.056 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:15c68a35:::benchmark_data_smithi138_490684_object52:head# 2024-09-15T05:51:39.057 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:16f818a6:::benchmark_data_smithi138_490684_object2:head# 2024-09-15T05:51:39.057 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:171e4ce8:::benchmark_data_smithi138_490684_object46:head# 2024-09-15T05:51:39.057 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:179c5a81:::benchmark_data_smithi138_490684_object249:head# 2024-09-15T05:51:39.058 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:18f581bb:::benchmark_data_smithi138_490684_object418:head# 2024-09-15T05:51:39.061 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:19adb72d:::benchmark_data_smithi138_490684_object8:head# 2024-09-15T05:51:39.061 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:19b8e577:::benchmark_data_smithi138_490684_object287:head# 2024-09-15T05:51:39.061 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:19d36c51:::benchmark_data_smithi138_490684_object327:head# 2024-09-15T05:51:39.063 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1a0f93bb:::benchmark_data_smithi138_490684_object192:head# 2024-09-15T05:51:39.063 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1a80306f:::benchmark_data_smithi138_490684_object218:head# 2024-09-15T05:51:39.064 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b45e6c9:::benchmark_data_smithi138_490684_object403:head# 2024-09-15T05:51:39.064 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1b760935:::benchmark_data_smithi138_490684_object404:head# 2024-09-15T05:51:39.065 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1c1d649a:::benchmark_data_smithi138_490684_object270:head# 2024-09-15T05:51:39.065 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1ccb86c9:::benchmark_data_smithi138_490684_object402:head# 2024-09-15T05:51:39.066 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1eeec1a0:::benchmark_data_smithi138_490684_object233:head# 2024-09-15T05:51:39.066 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1f6d0a3e:::benchmark_data_smithi138_490684_object151:head# 2024-09-15T05:51:39.067 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1fc18758:::benchmark_data_smithi138_490684_object336:head# 2024-09-15T05:51:39.067 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:1feaa880:::benchmark_data_smithi138_490684_object447:head# 2024-09-15T05:51:39.069 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:202f929b:::benchmark_data_smithi138_490684_object13:head# 2024-09-15T05:51:39.069 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2044cdd6:::benchmark_data_smithi138_490684_object439:head# 2024-09-15T05:51:39.069 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20c09b70:::benchmark_data_smithi138_490684_object220:head# 2024-09-15T05:51:39.070 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:20e43197:::benchmark_data_smithi138_490684_object307:head# 2024-09-15T05:51:39.070 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:234a43ee:::benchmark_data_smithi138_490684_object397:head# 2024-09-15T05:51:39.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:23661c4e:::benchmark_data_smithi138_490684_object342:head# 2024-09-15T05:51:39.071 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:23e07dde:::benchmark_data_smithi138_490684_object22:head# 2024-09-15T05:51:39.072 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:246b59bd:::benchmark_data_smithi138_490684_object103:head# 2024-09-15T05:51:39.072 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:247704b8:::benchmark_data_smithi138_490684_object303:head# 2024-09-15T05:51:39.075 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2518241a:::benchmark_data_smithi138_490684_object437:head# 2024-09-15T05:51:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:25492166:::benchmark_data_smithi138_490684_object391:head# 2024-09-15T05:51:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26941902:::benchmark_data_smithi138_490684_object49:head# 2024-09-15T05:51:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:269b6d63:::benchmark_data_smithi138_490684_object145:head# 2024-09-15T05:51:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:26e68a97:::benchmark_data_smithi138_490684_object306:head# 2024-09-15T05:51:39.076 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2779596d:::benchmark_data_smithi138_490684_object258:head# 2024-09-15T05:51:39.077 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:288a9553:::benchmark_data_smithi138_490684_object317:head# 2024-09-15T05:51:39.081 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:296cda35:::benchmark_data_smithi138_490684_object173:head# 2024-09-15T05:51:39.081 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:29f05fcf:::benchmark_data_smithi138_490684_object93:head# 2024-09-15T05:51:39.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2a0cb4d9:::benchmark_data_smithi138_490684_object291:head# 2024-09-15T05:51:39.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2abc5a35:::benchmark_data_smithi138_490684_object157:head# 2024-09-15T05:51:39.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b1aff40:::benchmark_data_smithi138_490684_object43:head# 2024-09-15T05:51:39.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b577641:::benchmark_data_smithi138_490684_object352:head# 2024-09-15T05:51:39.082 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b62c0ff:::benchmark_data_smithi138_490684_object228:head# 2024-09-15T05:51:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2b8013ae:::benchmark_data_smithi138_490684_object180:head# 2024-09-15T05:51:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2c17fc53:::benchmark_data_smithi138_490684_object415:head# 2024-09-15T05:51:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2e43a256:::benchmark_data_smithi138_490684_object167:head# 2024-09-15T05:51:39.083 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2e7d5de6:::benchmark_data_smithi138_490684_object115:head# 2024-09-15T05:51:39.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:2f223f31:::benchmark_data_smithi138_490684_object158:head# 2024-09-15T05:51:39.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3006c250:::benchmark_data_smithi138_490684_object77:head# 2024-09-15T05:51:39.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:304382b1:::benchmark_data_smithi138_490684_object244:head# 2024-09-15T05:51:39.089 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:30a692a5:::benchmark_data_smithi138_490684_object426:head# 2024-09-15T05:51:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:322a0e94:::benchmark_data_smithi138_490684_object143:head# 2024-09-15T05:51:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:33a85685:::benchmark_data_smithi138_490684_object87:head# 2024-09-15T05:51:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3436b730:::benchmark_data_smithi138_490684_object5:head# 2024-09-15T05:51:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3441a068:::benchmark_data_smithi138_490684_object42:head# 2024-09-15T05:51:39.090 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:353502aa:::benchmark_data_smithi138_490684_object372:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3564c248:::benchmark_data_smithi138_490684_object41:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:364aae3b:::benchmark_data_smithi138_490684_object110:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38c36887:::benchmark_data_smithi138_490684_object241:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:38d3be92:::benchmark_data_smithi138_490684_object205:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:39a6bdc7:::benchmark_data_smithi138_490684_object201:head# 2024-09-15T05:51:39.091 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:39cd9b03:::benchmark_data_smithi138_490684_object221:head# 2024-09-15T05:51:39.092 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3a86025a:::benchmark_data_smithi138_490684_object278:head# 2024-09-15T05:51:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3b387f0a:::benchmark_data_smithi138_490684_object60:head# 2024-09-15T05:51:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3b702768:::benchmark_data_smithi138_490684_object299:head# 2024-09-15T05:51:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3c45afcd:::benchmark_data_smithi138_490684_object204:head# 2024-09-15T05:51:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3cb02573:::benchmark_data_smithi138_490684_object324:head# 2024-09-15T05:51:39.098 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3ccd428b:::benchmark_data_smithi138_490684_object81:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3d073400:::benchmark_data_smithi138_490684_object385:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3e526c80:::benchmark_data_smithi138_490684_object310:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3f9cdcb2:::benchmark_data_smithi138_490684_object364:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:3fa970a1:::benchmark_data_smithi138_490684_object271:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:40ae5e8f:::benchmark_data_smithi138_490684_object135:head# 2024-09-15T05:51:39.099 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:40be8c6c:::benchmark_data_smithi138_490684_object400:head# 2024-09-15T05:51:39.100 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:41028aa4:::benchmark_data_smithi138_490684_object121:head# 2024-09-15T05:51:39.100 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:41e14e3c:::benchmark_data_smithi138_490684_object256:head# 2024-09-15T05:51:39.100 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:43a16967:::benchmark_data_smithi138_490684_object11:head# 2024-09-15T05:51:39.100 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:44c2271b:::benchmark_data_smithi138_490684_object165:head# 2024-09-15T05:51:39.100 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4524d44d:::benchmark_data_smithi138_490684_object416:head# 2024-09-15T05:51:39.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:45ab54b1:::benchmark_data_smithi138_490684_object253:head# 2024-09-15T05:51:39.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:46117dea:::benchmark_data_smithi138_490684_object374:head# 2024-09-15T05:51:39.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:467161af:::benchmark_data_smithi138_490684_object133:head# 2024-09-15T05:51:39.107 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48a397b7:::benchmark_data_smithi138_490684_object18:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:48d28611:::benchmark_data_smithi138_490684_object101:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:49c540fd:::benchmark_data_smithi138_490684_object126:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4a378d87:::benchmark_data_smithi138_490684_object102:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4b131268:::benchmark_data_smithi138_490684_object206:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4cce40a5:::benchmark_data_smithi138_490684_object252:head# 2024-09-15T05:51:39.108 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4e204145:::benchmark_data_smithi138_490684_object213:head# 2024-09-15T05:51:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:4edf41ef:::benchmark_data_smithi138_490684_object182:head# 2024-09-15T05:51:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:507de02a:::benchmark_data_smithi138_490684_object185:head# 2024-09-15T05:51:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:511cc58a:::benchmark_data_smithi138_490684_object363:head# 2024-09-15T05:51:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5122e41a:::benchmark_data_smithi138_490684_object129:head# 2024-09-15T05:51:39.109 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:514c8f50:::benchmark_data_smithi138_490684_object17:head# 2024-09-15T05:51:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:521c9aa3:::benchmark_data_smithi138_490684_object368:head# 2024-09-15T05:51:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5228d5ff:::benchmark_data_smithi138_490684_object225:head# 2024-09-15T05:51:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5285718c:::benchmark_data_smithi138_490684_object149:head# 2024-09-15T05:51:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:52fad99c:::benchmark_data_smithi138_490684_object94:head# 2024-09-15T05:51:39.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5405106b:::benchmark_data_smithi138_490684_object88:head# 2024-09-15T05:51:39.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:54ac2456:::benchmark_data_smithi138_490684_object261:head# 2024-09-15T05:51:39.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:55c76fad:::benchmark_data_smithi138_490684_object242:head# 2024-09-15T05:51:39.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:56916011:::benchmark_data_smithi138_490684_object263:head# 2024-09-15T05:51:39.122 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:574a1d45:::benchmark_data_smithi138_490684_object200:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:57a10393:::benchmark_data_smithi138_490684_object12:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:586914f6:::benchmark_data_smithi138_490684_object124:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:59c5126b:::benchmark_data_smithi138_490684_object321:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:59ce432d:::benchmark_data_smithi138_490684_object70:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:59fcba78:::benchmark_data_smithi138_490684_object45:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5a4fcb3c:::benchmark_data_smithi138_490684_object74:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5aa528d1:::benchmark_data_smithi138_490684_object290:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5b6d5834:::benchmark_data_smithi138_490684_object268:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5c37c848:::benchmark_data_smithi138_490684_object61:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5c6e05af:::benchmark_data_smithi138_490684_object331:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5c92f32d:::benchmark_data_smithi138_490684_object355:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:5f869df0:::benchmark_data_smithi138_490684_object232:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:604475f4:::benchmark_data_smithi138_490684_object97:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:60598111:::benchmark_data_smithi138_490684_object274:head# 2024-09-15T05:51:39.123 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:60599421:::benchmark_data_smithi138_490684_object305:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:60daded0:::benchmark_data_smithi138_490684_object184:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:61905448:::benchmark_data_smithi138_490684_object443:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:625a04ff:::benchmark_data_smithi138_490684_object195:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:62f23f1c:::benchmark_data_smithi138_490684_object224:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:631b20d2:::benchmark_data_smithi138_490684_object29:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6350ebfd:::benchmark_data_smithi138_490684_object419:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6473babf:::benchmark_data_smithi138_490684_object390:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:64b9cb32:::benchmark_data_smithi138_490684_object86:head# 2024-09-15T05:51:39.124 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:64c45162:::benchmark_data_smithi138_490684_object138:head# 2024-09-15T05:51:39.125 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:64f9efa8:::benchmark_data_smithi138_490684_object260:head# 2024-09-15T05:51:39.125 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:65484d91:::benchmark_data_smithi138_490684_object83:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:656e2b3b:::benchmark_data_smithi138_490684_object410:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6607bcb7:::benchmark_data_smithi138_490684_object219:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:66343d45:::benchmark_data_smithi138_490684_object240:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6637afbb:::benchmark_data_smithi138_490684_object392:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6730f283:::benchmark_data_smithi138_490684_object6:head# 2024-09-15T05:51:39.126 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:68f62b4d:::benchmark_data_smithi138_490684_object280:head# 2024-09-15T05:51:39.127 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:692362e5:::benchmark_data_smithi138_490684_object286:head# 2024-09-15T05:51:39.127 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:692416de:::benchmark_data_smithi138_490684_object127:head# 2024-09-15T05:51:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6b6fd3cc:::benchmark_data_smithi138_490684_object432:head# 2024-09-15T05:51:39.128 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6bc7ff99:::benchmark_data_smithi138_490684_object375:head# 2024-09-15T05:51:39.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6beab5ff:::benchmark_data_smithi138_490684_object176:head# 2024-09-15T05:51:39.129 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6c6ab043:::benchmark_data_smithi138_490684_object238:head# 2024-09-15T05:51:39.130 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6df45f4d:::benchmark_data_smithi138_490684_object28:head# 2024-09-15T05:51:39.130 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6dfdb530:::benchmark_data_smithi138_490684_object215:head# 2024-09-15T05:51:39.131 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:6fbb52d5:::benchmark_data_smithi138_490684_object172:head# 2024-09-15T05:51:39.131 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:727c9c31:::benchmark_data_smithi138_490684_object34:head# 2024-09-15T05:51:39.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:72dfdd50:::benchmark_data_smithi138_490684_object92:head# 2024-09-15T05:51:39.132 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:730f80da:::benchmark_data_smithi138_490684_object358:head# 2024-09-15T05:51:39.133 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:736ab456:::benchmark_data_smithi138_490684_object281:head# 2024-09-15T05:51:39.133 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:74dd6bcc:::benchmark_data_smithi138_490684_object433:head# 2024-09-15T05:51:39.133 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:756d5b13:::benchmark_data_smithi138_490684_object51:head# 2024-09-15T05:51:39.134 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7583884e:::benchmark_data_smithi138_490684_object301:head# 2024-09-15T05:51:39.134 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:76892cfa:::benchmark_data_smithi138_490684_object414:head# 2024-09-15T05:51:39.135 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:76dc975c:::benchmark_data_smithi138_490684_object367:head# 2024-09-15T05:51:39.136 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:770ddf44:::benchmark_data_smithi138_490684_object264:head# 2024-09-15T05:51:39.136 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:771cc600:::benchmark_data_smithi138_490684_object435:head# 2024-09-15T05:51:39.137 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:777b0415:::benchmark_data_smithi138_490684_object36:head# 2024-09-15T05:51:39.137 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:786d63d8:::benchmark_data_smithi138_490684_object160:head# 2024-09-15T05:51:39.138 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7943937d:::benchmark_data_smithi138_490684_object257:head# 2024-09-15T05:51:39.138 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:79bacdc0:::benchmark_data_smithi138_490684_object344:head# 2024-09-15T05:51:39.138 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:79ffad34:::benchmark_data_smithi138_490684_object169:head# 2024-09-15T05:51:39.139 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7ace6539:::benchmark_data_smithi138_490684_object328:head# 2024-09-15T05:51:39.140 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7c9bce91:::benchmark_data_smithi138_490684_object48:head# 2024-09-15T05:51:39.140 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d71f5eb:::benchmark_data_smithi138_490684_object78:head# 2024-09-15T05:51:39.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7d8c8279:::benchmark_data_smithi138_490684_object89:head# 2024-09-15T05:51:39.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7e1be448:::benchmark_data_smithi138_490684_object442:head# 2024-09-15T05:51:39.141 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:7fb18a34:::benchmark_data_smithi138_490684_object161:head# 2024-09-15T05:51:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:810ddcff:::benchmark_data_smithi138_490684_object296:head# 2024-09-15T05:51:39.142 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:815efc06:::benchmark_data_smithi138_490684_object117:head# 2024-09-15T05:51:39.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:817c4c83:::benchmark_data_smithi138_490684_object65:head# 2024-09-15T05:51:39.143 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8196b224:::benchmark_data_smithi138_490684_object338:head# 2024-09-15T05:51:39.144 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:81fe48d5:::benchmark_data_smithi138_490684_object430:head# 2024-09-15T05:51:39.144 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:825d552e:::benchmark_data_smithi138_490684_object272:head# 2024-09-15T05:51:39.145 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8303e10c:::benchmark_data_smithi138_490684_object277:head# 2024-09-15T05:51:39.145 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:831de136:::benchmark_data_smithi138_490684_object16:head# 2024-09-15T05:51:39.146 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:84fda897:::benchmark_data_smithi138_490684_object35:head# 2024-09-15T05:51:39.146 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:85a4a546:::benchmark_data_smithi138_490684_object186:head# 2024-09-15T05:51:39.147 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:871f0fc4:::benchmark_data_smithi138_490684_object67:head# 2024-09-15T05:51:39.147 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:88ce4ee6:::benchmark_data_smithi138_490684_object178:head# 2024-09-15T05:51:39.148 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:89845e71:::benchmark_data_smithi138_490684_object118:head# 2024-09-15T05:51:39.148 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8a2cd733:::benchmark_data_smithi138_490684_object322:head# 2024-09-15T05:51:39.149 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8a689f13:::benchmark_data_smithi138_490684_object316:head# 2024-09-15T05:51:39.150 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8ac7ba96:::benchmark_data_smithi138_490684_object387:head# 2024-09-15T05:51:39.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8b112df5:::benchmark_data_smithi138_490684_object199:head# 2024-09-15T05:51:39.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8c23723c:::benchmark_data_smithi138_490684_object152:head# 2024-09-15T05:51:39.151 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8c488088:::benchmark_data_smithi138_490684_object345:head# 2024-09-15T05:51:39.152 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8c92e977:::benchmark_data_smithi138_490684_object119:head# 2024-09-15T05:51:39.152 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8cc06d6f:::benchmark_data_smithi138_490684_object130:head# 2024-09-15T05:51:39.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8cf9b11f:::benchmark_data_smithi138_490684_object304:head# 2024-09-15T05:51:39.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d524128:::benchmark_data_smithi138_490684_object312:head# 2024-09-15T05:51:39.153 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d713c3a:::benchmark_data_smithi138_490684_object187:head# 2024-09-15T05:51:39.154 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d840dc6:::benchmark_data_smithi138_490684_object431:head# 2024-09-15T05:51:39.154 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8d957bf4:::benchmark_data_smithi138_490684_object350:head# 2024-09-15T05:51:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8dd0f22a:::benchmark_data_smithi138_490684_object222:head# 2024-09-15T05:51:39.155 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8e1de27b:::benchmark_data_smithi138_490684_object393:head# 2024-09-15T05:51:39.156 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:8f255c34:::benchmark_data_smithi138_490684_object82:head# 2024-09-15T05:51:39.157 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90586522:::benchmark_data_smithi138_490684_object259:head# 2024-09-15T05:51:39.157 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:90753c8a:::benchmark_data_smithi138_490684_object284:head# 2024-09-15T05:51:39.157 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:917a9c22:::benchmark_data_smithi138_490684_object379:head# 2024-09-15T05:51:39.158 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91b2dee2:::benchmark_data_smithi138_490684_object369:head# 2024-09-15T05:51:39.159 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91ddf18b:::benchmark_data_smithi138_490684_object114:head# 2024-09-15T05:51:39.159 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:91e7c8c6:::benchmark_data_smithi138_490684_object339:head# 2024-09-15T05:51:39.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:92303393:::benchmark_data_smithi138_490684_object420:head# 2024-09-15T05:51:39.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:955168dd:::benchmark_data_smithi138_490684_object136:head# 2024-09-15T05:51:39.160 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9688a187:::benchmark_data_smithi138_490684_object194:head# 2024-09-15T05:51:39.161 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:968957e2:::benchmark_data_smithi138_490684_object188:head# 2024-09-15T05:51:39.161 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:97adfced:::benchmark_data_smithi138_490684_object282:head# 2024-09-15T05:51:39.162 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:97e53fd1:::benchmark_data_smithi138_490684_object62:head# 2024-09-15T05:51:39.162 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:980bddc8:::benchmark_data_smithi138_490684_object111:head# 2024-09-15T05:51:39.163 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:982a5b5b:::benchmark_data_smithi138_490684_object356:head# 2024-09-15T05:51:39.163 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9902060d:::benchmark_data_smithi138_490684_object9:head# 2024-09-15T05:51:39.164 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:995dd824:::benchmark_data_smithi138_490684_object239:head# 2024-09-15T05:51:39.164 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9968873b:::benchmark_data_smithi138_490684_object255:head# 2024-09-15T05:51:39.164 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:99e016af:::benchmark_data_smithi138_490684_object3:head# 2024-09-15T05:51:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a1fcc1d:::benchmark_data_smithi138_490684_object332:head# 2024-09-15T05:51:39.165 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a451ff3:::benchmark_data_smithi138_490684_object309:head# 2024-09-15T05:51:39.166 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a5403b2:::benchmark_data_smithi138_490684_object15:head# 2024-09-15T05:51:39.166 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9a89159c:::benchmark_data_smithi138_490684_object214:head# 2024-09-15T05:51:39.167 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9b32f70a:::benchmark_data_smithi138_490684_object361:head# 2024-09-15T05:51:39.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9b5e0b9a:::benchmark_data_smithi138_490684_object434:head# 2024-09-15T05:51:39.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9bc94544:::benchmark_data_smithi138_490684_object230:head# 2024-09-15T05:51:39.168 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9bf085a4:::benchmark_data_smithi138_490684_object231:head# 2024-09-15T05:51:39.169 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9c11d375:::benchmark_data_smithi138_490684_object408:head# 2024-09-15T05:51:39.169 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9ced90eb:::benchmark_data_smithi138_490684_object223:head# 2024-09-15T05:51:39.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9d21bfdc:::benchmark_data_smithi138_490684_object1:head# 2024-09-15T05:51:39.170 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9d5eef2a:::benchmark_data_smithi138_490684_object106:head# 2024-09-15T05:51:39.171 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9d98a18a:::benchmark_data_smithi138_490684_object362:head# 2024-09-15T05:51:39.171 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9dec16a0:::benchmark_data_smithi138_490684_object140:head# 2024-09-15T05:51:39.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f1004b2:::benchmark_data_smithi138_490684_object371:head# 2024-09-15T05:51:39.172 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f8bac22:::benchmark_data_smithi138_490684_object100:head# 2024-09-15T05:51:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:9f8f3816:::benchmark_data_smithi138_490684_object386:head# 2024-09-15T05:51:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a02d48a2:::benchmark_data_smithi138_490684_object383:head# 2024-09-15T05:51:39.173 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0cca6ff:::benchmark_data_smithi138_490684_object285:head# 2024-09-15T05:51:39.174 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a0f7a3f1:::benchmark_data_smithi138_490684_object163:head# 2024-09-15T05:51:39.175 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a1239f7f:::benchmark_data_smithi138_490684_object198:head# 2024-09-15T05:51:39.175 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a24fa49c:::benchmark_data_smithi138_490684_object423:head# 2024-09-15T05:51:39.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a4fc753b:::benchmark_data_smithi138_490684_object394:head# 2024-09-15T05:51:39.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a55afec3:::benchmark_data_smithi138_490684_object250:head# 2024-09-15T05:51:39.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a604a5da:::benchmark_data_smithi138_490684_object348:head# 2024-09-15T05:51:39.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a72003a8:::benchmark_data_smithi138_490684_object212:head# 2024-09-15T05:51:39.177 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a748c773:::benchmark_data_smithi138_490684_object50:head# 2024-09-15T05:51:39.178 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a75235d1:::benchmark_data_smithi138_490684_object288:head# 2024-09-15T05:51:39.178 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a79fbd7e:::benchmark_data_smithi138_490684_object254:head# 2024-09-15T05:51:39.179 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a7a1198d:::benchmark_data_smithi138_490684_object128:head# 2024-09-15T05:51:39.179 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a7fc3f4e:::benchmark_data_smithi138_490684_object292:head# 2024-09-15T05:51:39.180 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a815fbd0:::benchmark_data_smithi138_490684_object349:head# 2024-09-15T05:51:39.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a829aa2a:::benchmark_data_smithi138_490684_object27:head# 2024-09-15T05:51:39.181 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a82ed670:::benchmark_data_smithi138_490684_object190:head# 2024-09-15T05:51:39.182 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:a9eab397:::benchmark_data_smithi138_490684_object347:head# 2024-09-15T05:51:39.182 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aa72d9bf:::benchmark_data_smithi138_490684_object131:head# 2024-09-15T05:51:39.182 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aa736f85:::benchmark_data_smithi138_490684_object210:head# 2024-09-15T05:51:39.183 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aae00d6a:::benchmark_data_smithi138_490684_object424:head# 2024-09-15T05:51:39.184 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ab691369:::benchmark_data_smithi138_490684_object171:head# 2024-09-15T05:51:39.184 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ac0f7c3f:::benchmark_data_smithi138_490684_object38:head# 2024-09-15T05:51:39.184 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:acae68aa:::benchmark_data_smithi138_490684_object373:head# 2024-09-15T05:51:39.185 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:acd3dc4e:::benchmark_data_smithi138_490684_object343:head# 2024-09-15T05:51:39.186 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ad7d3fda:::benchmark_data_smithi138_490684_object276:head# 2024-09-15T05:51:39.186 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ada7bb33:::benchmark_data_smithi138_490684_object323:head# 2024-09-15T05:51:39.186 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ae507d31:::benchmark_data_smithi138_490684_object142:head# 2024-09-15T05:51:39.187 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:aed39694:::benchmark_data_smithi138_490684_object360:head# 2024-09-15T05:51:39.187 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:afc9dbd4:::benchmark_data_smithi138_490684_object398:head# 2024-09-15T05:51:39.188 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:afd27c18:::benchmark_data_smithi138_490684_object139:head# 2024-09-15T05:51:39.189 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b188dc72:::benchmark_data_smithi138_490684_object245:head# 2024-09-15T05:51:39.189 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b19c4782:::benchmark_data_smithi138_490684_object203:head# 2024-09-15T05:51:39.190 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b1fc5c75:::benchmark_data_smithi138_490684_object174:head# 2024-09-15T05:51:39.190 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b34319c5:::benchmark_data_smithi138_490684_object427:head# 2024-09-15T05:51:39.190 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b4506c06:::benchmark_data_smithi138_490684_object40:head# 2024-09-15T05:51:39.191 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b502b170:::benchmark_data_smithi138_490684_object227:head# 2024-09-15T05:51:39.191 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b5979a85:::benchmark_data_smithi138_490684_object388:head# 2024-09-15T05:51:39.192 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b602263f:::benchmark_data_smithi138_490684_object300:head# 2024-09-15T05:51:39.192 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b6ace319:::benchmark_data_smithi138_490684_object53:head# 2024-09-15T05:51:39.193 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b712434d:::benchmark_data_smithi138_490684_object37:head# 2024-09-15T05:51:39.193 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b76c65e1:::benchmark_data_smithi138_490684_object325:head# 2024-09-15T05:51:39.194 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8033cc1:::benchmark_data_smithi138_490684_object76:head# 2024-09-15T05:51:39.194 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b81dbbdb:::benchmark_data_smithi138_490684_object318:head# 2024-09-15T05:51:39.195 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:51:39.195 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8bfddf1:::benchmark_data_smithi138_490684_object298:head# 2024-09-15T05:51:39.196 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:b8c141d6:::benchmark_data_smithi138_490684_object247:head# 2024-09-15T05:51:39.196 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ba027fe8:::benchmark_data_smithi138_490684_object314:head# 2024-09-15T05:51:39.197 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ba6b497c:::benchmark_data_smithi138_490684_object413:head# 2024-09-15T05:51:39.197 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ba974ce9:::benchmark_data_smithi138_490684_object251:head# 2024-09-15T05:51:39.198 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:baac4be2:::benchmark_data_smithi138_490684_object209:head# 2024-09-15T05:51:39.198 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:babd7411:::benchmark_data_smithi138_490684_object30:head# 2024-09-15T05:51:39.198 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bc183f22:::benchmark_data_smithi138_490684_object382:head# 2024-09-15T05:51:39.199 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bce033f2:::benchmark_data_smithi138_490684_object175:head# 2024-09-15T05:51:39.200 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bdddf4cd:::benchmark_data_smithi138_490684_object293:head# 2024-09-15T05:51:39.200 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:becb43cd:::benchmark_data_smithi138_490684_object267:head# 2024-09-15T05:51:39.201 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:bee75cac:::benchmark_data_smithi138_490684_object44:head# 2024-09-15T05:51:39.201 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c126efe4:::benchmark_data_smithi138_490684_object23:head# 2024-09-15T05:51:39.202 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c18b114d:::benchmark_data_smithi138_490684_object417:head# 2024-09-15T05:51:39.202 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c255df2d:::benchmark_data_smithi138_490684_object370:head# 2024-09-15T05:51:39.203 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c3cdf8b8:::benchmark_data_smithi138_490684_object302:head# 2024-09-15T05:51:39.203 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c41f5dd1:::benchmark_data_smithi138_490684_object20:head# 2024-09-15T05:51:39.204 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c48d081e:::benchmark_data_smithi138_490684_object116:head# 2024-09-15T05:51:39.204 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c4da0bf7:::benchmark_data_smithi138_490684_object146:head# 2024-09-15T05:51:39.204 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c587d3eb:::benchmark_data_smithi138_490684_object234:head# 2024-09-15T05:51:39.205 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c5affd17:::benchmark_data_smithi138_490684_object346:head# 2024-09-15T05:51:39.205 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c710ddf5:::benchmark_data_smithi138_490684_object235:head# 2024-09-15T05:51:39.206 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7abfafd:::benchmark_data_smithi138_490684_object69:head# 2024-09-15T05:51:39.206 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c7ec31b7:::benchmark_data_smithi138_490684_object193:head# 2024-09-15T05:51:39.207 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c859c77d:::benchmark_data_smithi138_490684_object297:head# 2024-09-15T05:51:39.207 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c871f5cb:::benchmark_data_smithi138_490684_object395:head# 2024-09-15T05:51:39.208 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8986227:::benchmark_data_smithi138_490684_object24:head# 2024-09-15T05:51:39.208 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c8ad720c:::benchmark_data_smithi138_490684_object150:head# 2024-09-15T05:51:39.209 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c9e075cc:::benchmark_data_smithi138_490684_object226:head# 2024-09-15T05:51:39.209 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:c9f27f6b:::benchmark_data_smithi138_490684_object207:head# 2024-09-15T05:51:39.210 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ca14c54c:::benchmark_data_smithi138_490684_object31:head# 2024-09-15T05:51:39.210 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ca74b47b:::benchmark_data_smithi138_490684_object381:head# 2024-09-15T05:51:39.210 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ca8ea2fa:::benchmark_data_smithi138_490684_object406:head# 2024-09-15T05:51:39.211 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cabfa736:::benchmark_data_smithi138_490684_object273:head# 2024-09-15T05:51:39.212 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cb047965:::benchmark_data_smithi138_490684_object401:head# 2024-09-15T05:51:39.212 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cbddb5da:::benchmark_data_smithi138_490684_object315:head# 2024-09-15T05:51:39.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cc6702b8:::benchmark_data_smithi138_490684_object265:head# 2024-09-15T05:51:39.213 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cc7bc92b:::benchmark_data_smithi138_490684_object217:head# 2024-09-15T05:51:39.214 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ccbad358:::benchmark_data_smithi138_490684_object262:head# 2024-09-15T05:51:39.215 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ccfaaa43:::benchmark_data_smithi138_490684_object269:head# 2024-09-15T05:51:39.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd0d6cbc:::benchmark_data_smithi138_490684_object25:head# 2024-09-15T05:51:39.216 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cd49f35b:::benchmark_data_smithi138_490684_object168:head# 2024-09-15T05:51:39.217 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:cf060d03:::benchmark_data_smithi138_490684_object354:head# 2024-09-15T05:51:39.217 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d11b26be:::benchmark_data_smithi138_490684_object68:head# 2024-09-15T05:51:39.217 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d311a17c:::benchmark_data_smithi138_490684_object154:head# 2024-09-15T05:51:39.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d317ea78:::benchmark_data_smithi138_490684_object19:head# 2024-09-15T05:51:39.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d45838c9:::benchmark_data_smithi138_490684_object295:head# 2024-09-15T05:51:39.218 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d561ea00:::benchmark_data_smithi138_490684_object440:head# 2024-09-15T05:51:39.219 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d5a23c35:::benchmark_data_smithi138_490684_object216:head# 2024-09-15T05:51:39.220 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d6d0b370:::benchmark_data_smithi138_490684_object75:head# 2024-09-15T05:51:39.220 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d6fd5d86:::benchmark_data_smithi138_490684_object66:head# 2024-09-15T05:51:39.221 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d7af5050:::benchmark_data_smithi138_490684_object153:head# 2024-09-15T05:51:39.221 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d878ba6f:::benchmark_data_smithi138_490684_object329:head# 2024-09-15T05:51:39.222 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8896898:::benchmark_data_smithi138_490684_object378:head# 2024-09-15T05:51:39.222 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d8ac82bf:::benchmark_data_smithi138_490684_object137:head# 2024-09-15T05:51:39.223 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d968376a:::benchmark_data_smithi138_490684_object441:head# 2024-09-15T05:51:39.223 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d9e0c3be:::benchmark_data_smithi138_490684_object320:head# 2024-09-15T05:51:39.224 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:d9fb1b48:::benchmark_data_smithi138_490684_object164:head# 2024-09-15T05:51:39.225 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:da4af231:::benchmark_data_smithi138_490684_object311:head# 2024-09-15T05:51:39.225 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:db2c59d2:::benchmark_data_smithi138_490684_object64:head# 2024-09-15T05:51:39.225 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:db3623e7:::benchmark_data_smithi138_490684_object0:head# 2024-09-15T05:51:39.226 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dbeebfbc:::benchmark_data_smithi138_490684_object425:head# 2024-09-15T05:51:39.226 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dccccbd3:::benchmark_data_smithi138_490684_object211:head# 2024-09-15T05:51:39.227 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dcf29248:::benchmark_data_smithi138_490684_object73:head# 2024-09-15T05:51:39.227 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd3cc6e6:::benchmark_data_smithi138_490684_object96:head# 2024-09-15T05:51:39.227 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd40e500:::benchmark_data_smithi138_490684_object39:head# 2024-09-15T05:51:39.228 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dd6a1e71:::benchmark_data_smithi138_490684_object279:head# 2024-09-15T05:51:39.228 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:de6a0aaa:::benchmark_data_smithi138_490684_object191:head# 2024-09-15T05:51:39.229 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:dea48036:::benchmark_data_smithi138_490684_object389:head# 2024-09-15T05:51:39.229 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:def79532:::benchmark_data_smithi138_490684_object308:head# 2024-09-15T05:51:39.230 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:df04885b:::benchmark_data_smithi138_490684_object99:head# 2024-09-15T05:51:39.230 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e07452fd:::benchmark_data_smithi138_490684_object335:head# 2024-09-15T05:51:39.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e0b05790:::benchmark_data_smithi138_490684_object85:head# 2024-09-15T05:51:39.231 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e0b656aa:::benchmark_data_smithi138_490684_object181:head# 2024-09-15T05:51:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e0e266eb:::benchmark_data_smithi138_490684_object112:head# 2024-09-15T05:51:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e106707e:::benchmark_data_smithi138_490684_object80:head# 2024-09-15T05:51:39.233 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e11daeb6:::benchmark_data_smithi138_490684_object341:head# 2024-09-15T05:51:39.233 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e14e303c:::benchmark_data_smithi138_490684_object366:head# 2024-09-15T05:51:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e2835e13:::benchmark_data_smithi138_490684_object71:head# 2024-09-15T05:51:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e2cb3a9c:::benchmark_data_smithi138_490684_object422:head# 2024-09-15T05:51:39.234 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e330ca21:::benchmark_data_smithi138_490684_object353:head# 2024-09-15T05:51:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e3827b88:::benchmark_data_smithi138_490684_object162:head# 2024-09-15T05:51:39.235 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e38bdbde:::benchmark_data_smithi138_490684_object429:head# 2024-09-15T05:51:39.236 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e39f10b6:::benchmark_data_smithi138_490684_object183:head# 2024-09-15T05:51:39.236 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e44b1845:::benchmark_data_smithi138_490684_object134:head# 2024-09-15T05:51:39.237 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e479c778:::benchmark_data_smithi138_490684_object365:head# 2024-09-15T05:51:39.237 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e4cd3c89:::benchmark_data_smithi138_490684_object144:head# 2024-09-15T05:51:39.238 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e604edd6:::benchmark_data_smithi138_490684_object32:head# 2024-09-15T05:51:39.238 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e658c1c4:::benchmark_data_smithi138_490684_object155:head# 2024-09-15T05:51:39.239 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e6881baa:::benchmark_data_smithi138_490684_object411:head# 2024-09-15T05:51:39.239 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e77e3c6d:::benchmark_data_smithi138_490684_object248:head# 2024-09-15T05:51:39.240 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e8354136:::benchmark_data_smithi138_490684_object376:head# 2024-09-15T05:51:39.240 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:e84b38d8:::benchmark_data_smithi138_490684_object26:head# 2024-09-15T05:51:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ea2a9c37:::benchmark_data_smithi138_490684_object444:head# 2024-09-15T05:51:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ea3a9c93:::benchmark_data_smithi138_490684_object330:head# 2024-09-15T05:51:39.242 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ea746610:::benchmark_data_smithi138_490684_object246:head# 2024-09-15T05:51:39.242 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:eaa02ef4:::benchmark_data_smithi138_490684_object72:head# 2024-09-15T05:51:39.243 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec164b11:::benchmark_data_smithi138_490684_object326:head# 2024-09-15T05:51:39.243 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec22600e:::benchmark_data_smithi138_490684_object294:head# 2024-09-15T05:51:39.244 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ec79d9e0:::benchmark_data_smithi138_490684_object108:head# 2024-09-15T05:51:39.244 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ecb3b783:::benchmark_data_smithi138_490684_object14:head# 2024-09-15T05:51:39.244 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ece76228:::benchmark_data_smithi138_490684_object313:head# 2024-09-15T05:51:39.245 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed25df26:::benchmark_data_smithi138_490684_object33:head# 2024-09-15T05:51:39.245 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ed850b4b:::benchmark_data_smithi138_490684_object428:head# 2024-09-15T05:51:39.246 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:edefdf02:::benchmark_data_smithi138_490684_object351:head# 2024-09-15T05:51:39.246 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ee044cee:::benchmark_data_smithi138_490684_object380:head# 2024-09-15T05:51:39.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ee80fec1:::benchmark_data_smithi138_490684_object202:head# 2024-09-15T05:51:39.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ef5377a3:::benchmark_data_smithi138_490684_object109:head# 2024-09-15T05:51:39.247 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:efac843c:::benchmark_data_smithi138_490684_object141:head# 2024-09-15T05:51:39.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:efc7cb10:::benchmark_data_smithi138_490684_object289:head# 2024-09-15T05:51:39.248 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:efe3dbc5:::benchmark_data_smithi138_490684_object132:head# 2024-09-15T05:51:39.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:efefe5df:::benchmark_data_smithi138_490684_object334:head# 2024-09-15T05:51:39.249 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f0198718:::benchmark_data_smithi138_490684_object170:head# 2024-09-15T05:51:39.250 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f0d693a5:::benchmark_data_smithi138_490684_object122:head# 2024-09-15T05:51:39.251 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f1880b91:::benchmark_data_smithi138_490684_object120:head# 2024-09-15T05:51:39.251 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f25e60ec:::benchmark_data_smithi138_490684_object359:head# 2024-09-15T05:51:39.252 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f2dc95dc:::benchmark_data_smithi138_490684_object229:head# 2024-09-15T05:51:39.252 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f347b8ae:::benchmark_data_smithi138_490684_object54:head# 2024-09-15T05:51:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f3c9e336:::benchmark_data_smithi138_490684_object283:head# 2024-09-15T05:51:39.253 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f3db25d5:::benchmark_data_smithi138_490684_object438:head# 2024-09-15T05:51:39.254 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f4244c3a:::benchmark_data_smithi138_490684_object208:head# 2024-09-15T05:51:39.254 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f55f7de3:::benchmark_data_smithi138_490684_object405:head# 2024-09-15T05:51:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f5bb8d1e:::benchmark_data_smithi138_490684_object123:head# 2024-09-15T05:51:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f619712b:::benchmark_data_smithi138_490684_object409:head# 2024-09-15T05:51:39.255 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f65a5eaa:::benchmark_data_smithi138_490684_object436:head# 2024-09-15T05:51:39.256 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f65f6043:::benchmark_data_smithi138_490684_object275:head# 2024-09-15T05:51:39.256 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f7c6c358:::benchmark_data_smithi138_490684_object337:head# 2024-09-15T05:51:39.257 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f8e1def2:::benchmark_data_smithi138_490684_object196:head# 2024-09-15T05:51:39.257 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f8f42a80:::benchmark_data_smithi138_490684_object446:head# 2024-09-15T05:51:39.258 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:f93bea36:::benchmark_data_smithi138_490684_object377:head# 2024-09-15T05:51:39.258 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fa473e4c:::benchmark_data_smithi138_490684_object125:head# 2024-09-15T05:51:39.259 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fb9decce:::benchmark_data_smithi138_490684_object396:head# 2024-09-15T05:51:39.260 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fbc6b792:::benchmark_data_smithi138_490684_object177:head# 2024-09-15T05:51:39.260 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fe5d71ab:::benchmark_data_smithi138_490684_object107:head# 2024-09-15T05:51:39.261 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:feb2f8f3:::benchmark_data_smithi138_490684_object421:head# 2024-09-15T05:51:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:fed63b8b:::benchmark_data_smithi138_490684_object90:head# 2024-09-15T05:51:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ff3c1513:::benchmark_data_smithi138_490684_object57:head# 2024-09-15T05:51:39.262 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ff75089e:::benchmark_data_smithi138_490684_object79:head# 2024-09-15T05:51:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffaa5f03:::benchmark_data_smithi138_490684_object147:head# 2024-09-15T05:51:39.263 INFO:tasks.workunit.client.0.smithi138.stderr:Read #1:ffcc0e7c:::benchmark_data_smithi138_490684_object412:head# 2024-09-15T05:51:39.264 INFO:tasks.workunit.client.0.smithi138.stderr:Export successful 2024-09-15T05:51:39.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-15T05:51:39.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:51:39.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:51:39.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:51:39.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:51:39.786 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:51:39.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:51:39.787 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:51:39.787 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-15T05:51:39.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:51:39.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:51:39.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:51:39.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:51:39.811 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:39.810+0000 7fdb2fd25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:39.813 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:39.812+0000 7fdb2fd25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:39.814 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:39.813+0000 7fdb2fd25440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:51:40.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:51:40.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:51:40.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:51:40.147 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:51:40.148 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:51:40.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:51:40.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:40.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:51:40.148 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:40.149 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:40.342+0000 7fdb2fd25440 -1 Falling back to public interface 2024-09-15T05:51:40.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:41.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:41.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:41.527 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:51:41.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:51:41.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:41.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:41.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:42.871 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:51:42.870+0000 7fdb2fd25440 -1 osd.0 9 log_to_monitors true 2024-09-15T05:51:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:42.885 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:51:42.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:51:42.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:42.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:43.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:51:44.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:51:44.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:51:44.251 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:51:44.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:51:44.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:51:44.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:51:44.637 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/491115,v1:127.0.0.1:6803/491115] [v2:127.0.0.1:6804/491115,v1:127.0.0.1:6805/491115] exists,up 32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:51:44.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:51:44.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:51:44.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:51:44.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-09-15T05:51:45.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-09-15T05:51:50.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-09-15T05:51:50.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-09-15T05:51:50.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-09-15T05:51:50.387 INFO:tasks.workunit.client.0.smithi138.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-09-15T05:51:50.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-09-15T05:51:50.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:51:50.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:51:50.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:51:50.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:51:50.389 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:51:50.389 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:51:50.389 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:51:50.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:51:50.390 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:51:50.477 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:51:50.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:51:50.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:51:50.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:51:50.907 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:51:51.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=51539607556 2024-09-15T05:51:51.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 51539607556 2024-09-15T05:51:51.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-51539607556' 2024-09-15T05:51:51.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:51:51.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-51539607556 2024-09-15T05:51:51.014 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:51:51.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:51:51.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-51539607556 2024-09-15T05:51:51.015 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:51:51.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=51539607556 2024-09-15T05:51:51.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-09-15T05:51:51.016 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 51539607556 2024-09-15T05:51:51.017 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:51:51.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607554 -lt 51539607556 2024-09-15T05:51:51.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:51:52.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:51:52.411 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:51:52.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 51539607556 -lt 51539607556 2024-09-15T05:51:52.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:51:52.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:51:52.775 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:51:53.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:51:53.264 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:51:53.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:51:53.604 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:51:53.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:51:53.605 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:51:54.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:51:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:51:54.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:51:54.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-09-15T05:51:55.451 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:51:55.451 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:51:55.451 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-09-15T05:51:55.767 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:51:56.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-15T05:51:57.280 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-09-15T05:51:57.280 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:51:57.281 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 1 -> 2 2024-09-15T05:51:57.281 INFO:tasks.workunit.client.0.smithi138.stderr: children are 1.1 2024-09-15T05:51:57.281 INFO:tasks.workunit.client.0.smithi138.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-09-15T05:51:57.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-09-15T05:51:58.559 INFO:tasks.workunit.client.0.smithi138.stdout: marking collection for removal 2024-09-15T05:51:58.559 INFO:tasks.workunit.client.0.smithi138.stdout:setting '_remove' omap key 2024-09-15T05:51:58.559 INFO:tasks.workunit.client.0.smithi138.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-09-15T05:51:59.043 INFO:tasks.workunit.client.0.smithi138.stdout:Remove successful 2024-09-15T05:51:59.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-09-15T05:52:00.271 INFO:tasks.workunit.client.0.smithi138.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-09-15T05:52:00.272 INFO:tasks.workunit.client.0.smithi138.stdout:Importing pgid 1.0 2024-09-15T05:52:00.272 INFO:tasks.workunit.client.0.smithi138.stdout:pool 1 pg_num 1 -> 2 2024-09-15T05:52:00.272 INFO:tasks.workunit.client.0.smithi138.stderr: children are 1.1 2024-09-15T05:52:00.273 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:004cdd36:::benchmark_data_smithi138_490684_object63:head# 2024-09-15T05:52:00.273 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.397 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:025d89ed:::benchmark_data_smithi138_490684_object59:head# 2024-09-15T05:52:00.397 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.447 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:035a4cdf:::benchmark_data_smithi138_490684_object47:head# 2024-09-15T05:52:00.447 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.497 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03afc33d:::benchmark_data_smithi138_490684_object319:head# 2024-09-15T05:52:00.497 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.543 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:03df6c74:::benchmark_data_smithi138_490684_object159:head# 2024-09-15T05:52:00.543 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.640 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:05e264b4:::benchmark_data_smithi138_490684_object445:head# 2024-09-15T05:52:00.640 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:00.966 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:060f17a0:::benchmark_data_smithi138_490684_object98:head# 2024-09-15T05:52:00.966 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.024 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:086e9542:::benchmark_data_smithi138_490684_object4:head# 2024-09-15T05:52:01.024 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.083 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0896a6d4:::benchmark_data_smithi138_490684_object166:head# 2024-09-15T05:52:01.083 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.141 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:08b73443:::benchmark_data_smithi138_490684_object148:head# 2024-09-15T05:52:01.141 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.200 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:09452e5b:::benchmark_data_smithi138_490684_object7:head# 2024-09-15T05:52:01.200 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.258 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0982f4db:::benchmark_data_smithi138_490684_object357:head# 2024-09-15T05:52:01.259 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.326 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a33106e:::benchmark_data_smithi138_490684_object179:head# 2024-09-15T05:52:01.326 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.401 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0a97a929:::benchmark_data_smithi138_490684_object84:head# 2024-09-15T05:52:01.401 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.459 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ac695c2:::benchmark_data_smithi138_490684_object384:head# 2024-09-15T05:52:01.459 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.518 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0ad5cfb3:::benchmark_data_smithi138_490684_object105:head# 2024-09-15T05:52:01.518 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.576 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0b6efcad:::benchmark_data_smithi138_490684_object95:head# 2024-09-15T05:52:01.577 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.610 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0c0d8acc:::benchmark_data_smithi138_490684_object21:head# 2024-09-15T05:52:01.610 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.644 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0c4fc3fa:::benchmark_data_smithi138_490684_object407:head# 2024-09-15T05:52:01.644 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.677 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0c5d3ae4:::benchmark_data_smithi138_490684_object399:head# 2024-09-15T05:52:01.677 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.710 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0d22a523:::benchmark_data_smithi138_490684_object243:head# 2024-09-15T05:52:01.710 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.744 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0d766a99:::benchmark_data_smithi138_490684_object55:head# 2024-09-15T05:52:01.744 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.777 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:0fcf5f0b:::benchmark_data_smithi138_490684_object237:head# 2024-09-15T05:52:01.777 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.811 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1079a170:::benchmark_data_smithi138_490684_object113:head# 2024-09-15T05:52:01.811 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.844 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10d8b0e9:::benchmark_data_smithi138_490684_object340:head# 2024-09-15T05:52:01.844 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.886 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10dda1b3:::benchmark_data_smithi138_490684_object236:head# 2024-09-15T05:52:01.886 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.920 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:10eb10a7:::benchmark_data_smithi138_490684_object104:head# 2024-09-15T05:52:01.920 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.953 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:118cebac:::benchmark_data_smithi138_490684_object189:head# 2024-09-15T05:52:01.953 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:01.995 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:12914623:::benchmark_data_smithi138_490684_object91:head# 2024-09-15T05:52:01.995 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.054 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:12e09e1e:::benchmark_data_smithi138_490684_object197:head# 2024-09-15T05:52:02.054 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.104 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:138b861d:::benchmark_data_smithi138_490684_object333:head# 2024-09-15T05:52:02.104 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.137 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14292895:::benchmark_data_smithi138_490684_object266:head# 2024-09-15T05:52:02.137 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.171 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14a6f7c5:::benchmark_data_smithi138_490684_object10:head# 2024-09-15T05:52:02.171 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.204 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:14cb2316:::benchmark_data_smithi138_490684_object58:head# 2024-09-15T05:52:02.204 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.254 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:158dca27:::benchmark_data_smithi138_490684_object56:head# 2024-09-15T05:52:02.254 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.288 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15907fb0:::benchmark_data_smithi138_490684_object156:head# 2024-09-15T05:52:02.288 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.321 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:15c68a35:::benchmark_data_smithi138_490684_object52:head# 2024-09-15T05:52:02.322 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.355 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:16f818a6:::benchmark_data_smithi138_490684_object2:head# 2024-09-15T05:52:02.355 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.388 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:171e4ce8:::benchmark_data_smithi138_490684_object46:head# 2024-09-15T05:52:02.388 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.422 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:179c5a81:::benchmark_data_smithi138_490684_object249:head# 2024-09-15T05:52:02.422 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.455 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:18f581bb:::benchmark_data_smithi138_490684_object418:head# 2024-09-15T05:52:02.455 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.497 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:19adb72d:::benchmark_data_smithi138_490684_object8:head# 2024-09-15T05:52:02.497 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.530 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:19b8e577:::benchmark_data_smithi138_490684_object287:head# 2024-09-15T05:52:02.530 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.589 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:19d36c51:::benchmark_data_smithi138_490684_object327:head# 2024-09-15T05:52:02.589 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.639 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1a0f93bb:::benchmark_data_smithi138_490684_object192:head# 2024-09-15T05:52:02.640 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.698 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1a80306f:::benchmark_data_smithi138_490684_object218:head# 2024-09-15T05:52:02.698 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.731 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b45e6c9:::benchmark_data_smithi138_490684_object403:head# 2024-09-15T05:52:02.732 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.773 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1b760935:::benchmark_data_smithi138_490684_object404:head# 2024-09-15T05:52:02.773 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.798 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1c1d649a:::benchmark_data_smithi138_490684_object270:head# 2024-09-15T05:52:02.798 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.824 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1ccb86c9:::benchmark_data_smithi138_490684_object402:head# 2024-09-15T05:52:02.824 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1eeec1a0:::benchmark_data_smithi138_490684_object233:head# 2024-09-15T05:52:02.849 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.874 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1f6d0a3e:::benchmark_data_smithi138_490684_object151:head# 2024-09-15T05:52:02.874 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.899 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1fc18758:::benchmark_data_smithi138_490684_object336:head# 2024-09-15T05:52:02.899 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.924 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:1feaa880:::benchmark_data_smithi138_490684_object447:head# 2024-09-15T05:52:02.924 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.949 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:202f929b:::benchmark_data_smithi138_490684_object13:head# 2024-09-15T05:52:02.949 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:02.983 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2044cdd6:::benchmark_data_smithi138_490684_object439:head# 2024-09-15T05:52:02.983 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.016 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20c09b70:::benchmark_data_smithi138_490684_object220:head# 2024-09-15T05:52:03.016 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.050 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:20e43197:::benchmark_data_smithi138_490684_object307:head# 2024-09-15T05:52:03.050 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.083 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:234a43ee:::benchmark_data_smithi138_490684_object397:head# 2024-09-15T05:52:03.083 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.117 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:23661c4e:::benchmark_data_smithi138_490684_object342:head# 2024-09-15T05:52:03.117 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.150 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:23e07dde:::benchmark_data_smithi138_490684_object22:head# 2024-09-15T05:52:03.150 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.183 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:246b59bd:::benchmark_data_smithi138_490684_object103:head# 2024-09-15T05:52:03.184 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.217 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:247704b8:::benchmark_data_smithi138_490684_object303:head# 2024-09-15T05:52:03.217 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.250 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2518241a:::benchmark_data_smithi138_490684_object437:head# 2024-09-15T05:52:03.251 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.284 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:25492166:::benchmark_data_smithi138_490684_object391:head# 2024-09-15T05:52:03.284 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.317 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26941902:::benchmark_data_smithi138_490684_object49:head# 2024-09-15T05:52:03.318 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.351 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:269b6d63:::benchmark_data_smithi138_490684_object145:head# 2024-09-15T05:52:03.351 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.384 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:26e68a97:::benchmark_data_smithi138_490684_object306:head# 2024-09-15T05:52:03.384 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.418 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2779596d:::benchmark_data_smithi138_490684_object258:head# 2024-09-15T05:52:03.418 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.451 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:288a9553:::benchmark_data_smithi138_490684_object317:head# 2024-09-15T05:52:03.451 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.485 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:296cda35:::benchmark_data_smithi138_490684_object173:head# 2024-09-15T05:52:03.485 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.518 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:29f05fcf:::benchmark_data_smithi138_490684_object93:head# 2024-09-15T05:52:03.518 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.552 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2a0cb4d9:::benchmark_data_smithi138_490684_object291:head# 2024-09-15T05:52:03.552 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.585 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2abc5a35:::benchmark_data_smithi138_490684_object157:head# 2024-09-15T05:52:03.585 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.619 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b1aff40:::benchmark_data_smithi138_490684_object43:head# 2024-09-15T05:52:03.619 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.652 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b577641:::benchmark_data_smithi138_490684_object352:head# 2024-09-15T05:52:03.652 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.685 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b62c0ff:::benchmark_data_smithi138_490684_object228:head# 2024-09-15T05:52:03.686 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.719 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2b8013ae:::benchmark_data_smithi138_490684_object180:head# 2024-09-15T05:52:03.719 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.752 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2c17fc53:::benchmark_data_smithi138_490684_object415:head# 2024-09-15T05:52:03.753 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.786 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2e43a256:::benchmark_data_smithi138_490684_object167:head# 2024-09-15T05:52:03.786 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.828 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2e7d5de6:::benchmark_data_smithi138_490684_object115:head# 2024-09-15T05:52:03.828 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.861 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:2f223f31:::benchmark_data_smithi138_490684_object158:head# 2024-09-15T05:52:03.862 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.895 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3006c250:::benchmark_data_smithi138_490684_object77:head# 2024-09-15T05:52:03.895 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.928 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:304382b1:::benchmark_data_smithi138_490684_object244:head# 2024-09-15T05:52:03.929 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.962 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:30a692a5:::benchmark_data_smithi138_490684_object426:head# 2024-09-15T05:52:03.962 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:03.995 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:322a0e94:::benchmark_data_smithi138_490684_object143:head# 2024-09-15T05:52:03.996 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.046 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:33a85685:::benchmark_data_smithi138_490684_object87:head# 2024-09-15T05:52:04.046 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.079 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3436b730:::benchmark_data_smithi138_490684_object5:head# 2024-09-15T05:52:04.079 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.113 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3441a068:::benchmark_data_smithi138_490684_object42:head# 2024-09-15T05:52:04.113 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.146 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:353502aa:::benchmark_data_smithi138_490684_object372:head# 2024-09-15T05:52:04.146 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.179 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3564c248:::benchmark_data_smithi138_490684_object41:head# 2024-09-15T05:52:04.180 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.213 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:364aae3b:::benchmark_data_smithi138_490684_object110:head# 2024-09-15T05:52:04.213 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.246 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38c36887:::benchmark_data_smithi138_490684_object241:head# 2024-09-15T05:52:04.246 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.296 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:38d3be92:::benchmark_data_smithi138_490684_object205:head# 2024-09-15T05:52:04.297 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.330 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39a6bdc7:::benchmark_data_smithi138_490684_object201:head# 2024-09-15T05:52:04.330 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.363 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:39cd9b03:::benchmark_data_smithi138_490684_object221:head# 2024-09-15T05:52:04.364 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.397 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3a86025a:::benchmark_data_smithi138_490684_object278:head# 2024-09-15T05:52:04.397 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.431 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3b387f0a:::benchmark_data_smithi138_490684_object60:head# 2024-09-15T05:52:04.431 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.464 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3b702768:::benchmark_data_smithi138_490684_object299:head# 2024-09-15T05:52:04.464 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.497 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3c45afcd:::benchmark_data_smithi138_490684_object204:head# 2024-09-15T05:52:04.497 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.531 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3cb02573:::benchmark_data_smithi138_490684_object324:head# 2024-09-15T05:52:04.531 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.589 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3ccd428b:::benchmark_data_smithi138_490684_object81:head# 2024-09-15T05:52:04.590 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.640 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3d073400:::benchmark_data_smithi138_490684_object385:head# 2024-09-15T05:52:04.640 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.698 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3e526c80:::benchmark_data_smithi138_490684_object310:head# 2024-09-15T05:52:04.699 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.724 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3f9cdcb2:::benchmark_data_smithi138_490684_object364:head# 2024-09-15T05:52:04.724 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.757 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:3fa970a1:::benchmark_data_smithi138_490684_object271:head# 2024-09-15T05:52:04.757 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.813 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:40ae5e8f:::benchmark_data_smithi138_490684_object135:head# 2024-09-15T05:52:04.814 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:40be8c6c:::benchmark_data_smithi138_490684_object400:head# 2024-09-15T05:52:04.849 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.883 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:41028aa4:::benchmark_data_smithi138_490684_object121:head# 2024-09-15T05:52:04.883 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.916 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:41e14e3c:::benchmark_data_smithi138_490684_object256:head# 2024-09-15T05:52:04.916 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.958 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:43a16967:::benchmark_data_smithi138_490684_object11:head# 2024-09-15T05:52:04.958 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:04.983 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:44c2271b:::benchmark_data_smithi138_490684_object165:head# 2024-09-15T05:52:04.983 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.008 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4524d44d:::benchmark_data_smithi138_490684_object416:head# 2024-09-15T05:52:05.008 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.033 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:45ab54b1:::benchmark_data_smithi138_490684_object253:head# 2024-09-15T05:52:05.033 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.058 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:46117dea:::benchmark_data_smithi138_490684_object374:head# 2024-09-15T05:52:05.058 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.083 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:467161af:::benchmark_data_smithi138_490684_object133:head# 2024-09-15T05:52:05.084 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.109 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48a397b7:::benchmark_data_smithi138_490684_object18:head# 2024-09-15T05:52:05.109 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.150 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:48d28611:::benchmark_data_smithi138_490684_object101:head# 2024-09-15T05:52:05.150 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.192 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:49c540fd:::benchmark_data_smithi138_490684_object126:head# 2024-09-15T05:52:05.192 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.226 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4a378d87:::benchmark_data_smithi138_490684_object102:head# 2024-09-15T05:52:05.226 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.259 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4b131268:::benchmark_data_smithi138_490684_object206:head# 2024-09-15T05:52:05.259 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.292 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4cce40a5:::benchmark_data_smithi138_490684_object252:head# 2024-09-15T05:52:05.293 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.326 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4e204145:::benchmark_data_smithi138_490684_object213:head# 2024-09-15T05:52:05.326 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.359 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:4edf41ef:::benchmark_data_smithi138_490684_object182:head# 2024-09-15T05:52:05.360 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.393 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:507de02a:::benchmark_data_smithi138_490684_object185:head# 2024-09-15T05:52:05.393 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.426 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:511cc58a:::benchmark_data_smithi138_490684_object363:head# 2024-09-15T05:52:05.427 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.460 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5122e41a:::benchmark_data_smithi138_490684_object129:head# 2024-09-15T05:52:05.460 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.493 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:514c8f50:::benchmark_data_smithi138_490684_object17:head# 2024-09-15T05:52:05.493 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.537 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:521c9aa3:::benchmark_data_smithi138_490684_object368:head# 2024-09-15T05:52:05.537 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.568 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5228d5ff:::benchmark_data_smithi138_490684_object225:head# 2024-09-15T05:52:05.569 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.602 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5285718c:::benchmark_data_smithi138_490684_object149:head# 2024-09-15T05:52:05.602 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.635 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:52fad99c:::benchmark_data_smithi138_490684_object94:head# 2024-09-15T05:52:05.636 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.669 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5405106b:::benchmark_data_smithi138_490684_object88:head# 2024-09-15T05:52:05.669 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.702 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:54ac2456:::benchmark_data_smithi138_490684_object261:head# 2024-09-15T05:52:05.703 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.736 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:55c76fad:::benchmark_data_smithi138_490684_object242:head# 2024-09-15T05:52:05.736 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.778 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:56916011:::benchmark_data_smithi138_490684_object263:head# 2024-09-15T05:52:05.778 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.811 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:574a1d45:::benchmark_data_smithi138_490684_object200:head# 2024-09-15T05:52:05.811 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.845 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:57a10393:::benchmark_data_smithi138_490684_object12:head# 2024-09-15T05:52:05.845 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.878 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:586914f6:::benchmark_data_smithi138_490684_object124:head# 2024-09-15T05:52:05.878 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.912 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:59c5126b:::benchmark_data_smithi138_490684_object321:head# 2024-09-15T05:52:05.912 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.945 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:59ce432d:::benchmark_data_smithi138_490684_object70:head# 2024-09-15T05:52:05.945 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:05.979 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:59fcba78:::benchmark_data_smithi138_490684_object45:head# 2024-09-15T05:52:05.979 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.029 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5a4fcb3c:::benchmark_data_smithi138_490684_object74:head# 2024-09-15T05:52:06.030 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.079 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5aa528d1:::benchmark_data_smithi138_490684_object290:head# 2024-09-15T05:52:06.079 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.113 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5b6d5834:::benchmark_data_smithi138_490684_object268:head# 2024-09-15T05:52:06.113 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.146 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c37c848:::benchmark_data_smithi138_490684_object61:head# 2024-09-15T05:52:06.146 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.179 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c6e05af:::benchmark_data_smithi138_490684_object331:head# 2024-09-15T05:52:06.180 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.213 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5c92f32d:::benchmark_data_smithi138_490684_object355:head# 2024-09-15T05:52:06.213 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.246 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:5f869df0:::benchmark_data_smithi138_490684_object232:head# 2024-09-15T05:52:06.246 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.280 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:604475f4:::benchmark_data_smithi138_490684_object97:head# 2024-09-15T05:52:06.280 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.313 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:60598111:::benchmark_data_smithi138_490684_object274:head# 2024-09-15T05:52:06.313 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.347 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:60599421:::benchmark_data_smithi138_490684_object305:head# 2024-09-15T05:52:06.347 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.381 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:60daded0:::benchmark_data_smithi138_490684_object184:head# 2024-09-15T05:52:06.381 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.414 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:61905448:::benchmark_data_smithi138_490684_object443:head# 2024-09-15T05:52:06.414 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.447 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:625a04ff:::benchmark_data_smithi138_490684_object195:head# 2024-09-15T05:52:06.447 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.481 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:62f23f1c:::benchmark_data_smithi138_490684_object224:head# 2024-09-15T05:52:06.481 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.514 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:631b20d2:::benchmark_data_smithi138_490684_object29:head# 2024-09-15T05:52:06.514 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.558 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6350ebfd:::benchmark_data_smithi138_490684_object419:head# 2024-09-15T05:52:06.559 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.606 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6473babf:::benchmark_data_smithi138_490684_object390:head# 2024-09-15T05:52:06.606 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.665 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64b9cb32:::benchmark_data_smithi138_490684_object86:head# 2024-09-15T05:52:06.665 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.716 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64c45162:::benchmark_data_smithi138_490684_object138:head# 2024-09-15T05:52:06.716 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.749 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:64f9efa8:::benchmark_data_smithi138_490684_object260:head# 2024-09-15T05:52:06.749 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.782 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:65484d91:::benchmark_data_smithi138_490684_object83:head# 2024-09-15T05:52:06.783 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.816 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:656e2b3b:::benchmark_data_smithi138_490684_object410:head# 2024-09-15T05:52:06.816 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.841 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6607bcb7:::benchmark_data_smithi138_490684_object219:head# 2024-09-15T05:52:06.841 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.874 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:66343d45:::benchmark_data_smithi138_490684_object240:head# 2024-09-15T05:52:06.874 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.899 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6637afbb:::benchmark_data_smithi138_490684_object392:head# 2024-09-15T05:52:06.900 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.925 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6730f283:::benchmark_data_smithi138_490684_object6:head# 2024-09-15T05:52:06.925 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.950 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:68f62b4d:::benchmark_data_smithi138_490684_object280:head# 2024-09-15T05:52:06.950 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:06.977 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:692362e5:::benchmark_data_smithi138_490684_object286:head# 2024-09-15T05:52:06.977 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.000 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:692416de:::benchmark_data_smithi138_490684_object127:head# 2024-09-15T05:52:07.000 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.025 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6b6fd3cc:::benchmark_data_smithi138_490684_object432:head# 2024-09-15T05:52:07.025 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.059 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6bc7ff99:::benchmark_data_smithi138_490684_object375:head# 2024-09-15T05:52:07.059 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.092 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6beab5ff:::benchmark_data_smithi138_490684_object176:head# 2024-09-15T05:52:07.092 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.125 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6c6ab043:::benchmark_data_smithi138_490684_object238:head# 2024-09-15T05:52:07.125 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.159 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6df45f4d:::benchmark_data_smithi138_490684_object28:head# 2024-09-15T05:52:07.159 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.192 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6dfdb530:::benchmark_data_smithi138_490684_object215:head# 2024-09-15T05:52:07.193 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.234 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:6fbb52d5:::benchmark_data_smithi138_490684_object172:head# 2024-09-15T05:52:07.234 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.268 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:727c9c31:::benchmark_data_smithi138_490684_object34:head# 2024-09-15T05:52:07.268 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.293 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:72dfdd50:::benchmark_data_smithi138_490684_object92:head# 2024-09-15T05:52:07.293 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.326 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:730f80da:::benchmark_data_smithi138_490684_object358:head# 2024-09-15T05:52:07.326 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.351 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:736ab456:::benchmark_data_smithi138_490684_object281:head# 2024-09-15T05:52:07.351 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.376 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:74dd6bcc:::benchmark_data_smithi138_490684_object433:head# 2024-09-15T05:52:07.376 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.410 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:756d5b13:::benchmark_data_smithi138_490684_object51:head# 2024-09-15T05:52:07.410 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.443 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7583884e:::benchmark_data_smithi138_490684_object301:head# 2024-09-15T05:52:07.444 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.477 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:76892cfa:::benchmark_data_smithi138_490684_object414:head# 2024-09-15T05:52:07.477 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.510 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:76dc975c:::benchmark_data_smithi138_490684_object367:head# 2024-09-15T05:52:07.510 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.544 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:770ddf44:::benchmark_data_smithi138_490684_object264:head# 2024-09-15T05:52:07.544 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.577 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:771cc600:::benchmark_data_smithi138_490684_object435:head# 2024-09-15T05:52:07.577 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.611 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:777b0415:::benchmark_data_smithi138_490684_object36:head# 2024-09-15T05:52:07.611 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.653 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:786d63d8:::benchmark_data_smithi138_490684_object160:head# 2024-09-15T05:52:07.653 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.686 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7943937d:::benchmark_data_smithi138_490684_object257:head# 2024-09-15T05:52:07.686 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.728 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:79bacdc0:::benchmark_data_smithi138_490684_object344:head# 2024-09-15T05:52:07.728 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.761 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:79ffad34:::benchmark_data_smithi138_490684_object169:head# 2024-09-15T05:52:07.762 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.795 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7ace6539:::benchmark_data_smithi138_490684_object328:head# 2024-09-15T05:52:07.795 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.829 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7c9bce91:::benchmark_data_smithi138_490684_object48:head# 2024-09-15T05:52:07.829 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.862 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d71f5eb:::benchmark_data_smithi138_490684_object78:head# 2024-09-15T05:52:07.862 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.895 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7d8c8279:::benchmark_data_smithi138_490684_object89:head# 2024-09-15T05:52:07.895 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.929 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7e1be448:::benchmark_data_smithi138_490684_object442:head# 2024-09-15T05:52:07.929 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:07.962 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:7fb18a34:::benchmark_data_smithi138_490684_object161:head# 2024-09-15T05:52:07.962 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.013 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:810ddcff:::benchmark_data_smithi138_490684_object296:head# 2024-09-15T05:52:08.013 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.063 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:815efc06:::benchmark_data_smithi138_490684_object117:head# 2024-09-15T05:52:08.063 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.096 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:817c4c83:::benchmark_data_smithi138_490684_object65:head# 2024-09-15T05:52:08.096 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.130 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8196b224:::benchmark_data_smithi138_490684_object338:head# 2024-09-15T05:52:08.130 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.163 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:81fe48d5:::benchmark_data_smithi138_490684_object430:head# 2024-09-15T05:52:08.163 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.197 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:825d552e:::benchmark_data_smithi138_490684_object272:head# 2024-09-15T05:52:08.197 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.230 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8303e10c:::benchmark_data_smithi138_490684_object277:head# 2024-09-15T05:52:08.230 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.263 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:831de136:::benchmark_data_smithi138_490684_object16:head# 2024-09-15T05:52:08.264 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.297 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:84fda897:::benchmark_data_smithi138_490684_object35:head# 2024-09-15T05:52:08.297 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.330 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:85a4a546:::benchmark_data_smithi138_490684_object186:head# 2024-09-15T05:52:08.331 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.364 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:871f0fc4:::benchmark_data_smithi138_490684_object67:head# 2024-09-15T05:52:08.364 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.397 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:88ce4ee6:::benchmark_data_smithi138_490684_object178:head# 2024-09-15T05:52:08.398 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.431 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:89845e71:::benchmark_data_smithi138_490684_object118:head# 2024-09-15T05:52:08.431 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.464 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a2cd733:::benchmark_data_smithi138_490684_object322:head# 2024-09-15T05:52:08.465 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.498 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8a689f13:::benchmark_data_smithi138_490684_object316:head# 2024-09-15T05:52:08.498 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.531 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8ac7ba96:::benchmark_data_smithi138_490684_object387:head# 2024-09-15T05:52:08.531 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.590 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8b112df5:::benchmark_data_smithi138_490684_object199:head# 2024-09-15T05:52:08.590 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.674 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c23723c:::benchmark_data_smithi138_490684_object152:head# 2024-09-15T05:52:08.675 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.749 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c488088:::benchmark_data_smithi138_490684_object345:head# 2024-09-15T05:52:08.749 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.783 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8c92e977:::benchmark_data_smithi138_490684_object119:head# 2024-09-15T05:52:08.783 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.816 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8cc06d6f:::benchmark_data_smithi138_490684_object130:head# 2024-09-15T05:52:08.817 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.849 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8cf9b11f:::benchmark_data_smithi138_490684_object304:head# 2024-09-15T05:52:08.850 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.883 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d524128:::benchmark_data_smithi138_490684_object312:head# 2024-09-15T05:52:08.883 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.917 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d713c3a:::benchmark_data_smithi138_490684_object187:head# 2024-09-15T05:52:08.917 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.950 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d840dc6:::benchmark_data_smithi138_490684_object431:head# 2024-09-15T05:52:08.950 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:08.984 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8d957bf4:::benchmark_data_smithi138_490684_object350:head# 2024-09-15T05:52:08.984 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.017 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8dd0f22a:::benchmark_data_smithi138_490684_object222:head# 2024-09-15T05:52:09.017 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.051 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8e1de27b:::benchmark_data_smithi138_490684_object393:head# 2024-09-15T05:52:09.051 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.076 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:8f255c34:::benchmark_data_smithi138_490684_object82:head# 2024-09-15T05:52:09.076 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.101 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90586522:::benchmark_data_smithi138_490684_object259:head# 2024-09-15T05:52:09.101 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.125 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:90753c8a:::benchmark_data_smithi138_490684_object284:head# 2024-09-15T05:52:09.126 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.151 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:917a9c22:::benchmark_data_smithi138_490684_object379:head# 2024-09-15T05:52:09.151 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.176 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91b2dee2:::benchmark_data_smithi138_490684_object369:head# 2024-09-15T05:52:09.176 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.201 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91ddf18b:::benchmark_data_smithi138_490684_object114:head# 2024-09-15T05:52:09.201 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.226 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:91e7c8c6:::benchmark_data_smithi138_490684_object339:head# 2024-09-15T05:52:09.226 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.251 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:92303393:::benchmark_data_smithi138_490684_object420:head# 2024-09-15T05:52:09.251 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.276 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:955168dd:::benchmark_data_smithi138_490684_object136:head# 2024-09-15T05:52:09.277 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.310 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9688a187:::benchmark_data_smithi138_490684_object194:head# 2024-09-15T05:52:09.310 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.343 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:968957e2:::benchmark_data_smithi138_490684_object188:head# 2024-09-15T05:52:09.343 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.377 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:97adfced:::benchmark_data_smithi138_490684_object282:head# 2024-09-15T05:52:09.377 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.410 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:97e53fd1:::benchmark_data_smithi138_490684_object62:head# 2024-09-15T05:52:09.410 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.444 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:980bddc8:::benchmark_data_smithi138_490684_object111:head# 2024-09-15T05:52:09.444 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.477 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:982a5b5b:::benchmark_data_smithi138_490684_object356:head# 2024-09-15T05:52:09.477 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.511 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9902060d:::benchmark_data_smithi138_490684_object9:head# 2024-09-15T05:52:09.511 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.544 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:995dd824:::benchmark_data_smithi138_490684_object239:head# 2024-09-15T05:52:09.544 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.586 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9968873b:::benchmark_data_smithi138_490684_object255:head# 2024-09-15T05:52:09.586 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.620 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:99e016af:::benchmark_data_smithi138_490684_object3:head# 2024-09-15T05:52:09.620 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.653 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a1fcc1d:::benchmark_data_smithi138_490684_object332:head# 2024-09-15T05:52:09.653 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.687 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a451ff3:::benchmark_data_smithi138_490684_object309:head# 2024-09-15T05:52:09.687 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.720 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a5403b2:::benchmark_data_smithi138_490684_object15:head# 2024-09-15T05:52:09.720 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.753 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9a89159c:::benchmark_data_smithi138_490684_object214:head# 2024-09-15T05:52:09.753 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9b32f70a:::benchmark_data_smithi138_490684_object361:head# 2024-09-15T05:52:09.787 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.821 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9b5e0b9a:::benchmark_data_smithi138_490684_object434:head# 2024-09-15T05:52:09.821 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.854 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9bc94544:::benchmark_data_smithi138_490684_object230:head# 2024-09-15T05:52:09.854 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.888 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9bf085a4:::benchmark_data_smithi138_490684_object231:head# 2024-09-15T05:52:09.888 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.921 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9c11d375:::benchmark_data_smithi138_490684_object408:head# 2024-09-15T05:52:09.921 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:09.963 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9ced90eb:::benchmark_data_smithi138_490684_object223:head# 2024-09-15T05:52:09.963 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.013 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9d21bfdc:::benchmark_data_smithi138_490684_object1:head# 2024-09-15T05:52:10.013 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.063 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9d5eef2a:::benchmark_data_smithi138_490684_object106:head# 2024-09-15T05:52:10.063 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.097 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9d98a18a:::benchmark_data_smithi138_490684_object362:head# 2024-09-15T05:52:10.097 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.130 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9dec16a0:::benchmark_data_smithi138_490684_object140:head# 2024-09-15T05:52:10.130 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.164 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f1004b2:::benchmark_data_smithi138_490684_object371:head# 2024-09-15T05:52:10.164 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.197 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f8bac22:::benchmark_data_smithi138_490684_object100:head# 2024-09-15T05:52:10.197 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.231 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:9f8f3816:::benchmark_data_smithi138_490684_object386:head# 2024-09-15T05:52:10.231 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.264 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a02d48a2:::benchmark_data_smithi138_490684_object383:head# 2024-09-15T05:52:10.264 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.314 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0cca6ff:::benchmark_data_smithi138_490684_object285:head# 2024-09-15T05:52:10.314 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.348 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a0f7a3f1:::benchmark_data_smithi138_490684_object163:head# 2024-09-15T05:52:10.348 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.381 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a1239f7f:::benchmark_data_smithi138_490684_object198:head# 2024-09-15T05:52:10.381 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.414 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a24fa49c:::benchmark_data_smithi138_490684_object423:head# 2024-09-15T05:52:10.415 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.448 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a4fc753b:::benchmark_data_smithi138_490684_object394:head# 2024-09-15T05:52:10.448 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.481 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a55afec3:::benchmark_data_smithi138_490684_object250:head# 2024-09-15T05:52:10.482 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.515 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a604a5da:::benchmark_data_smithi138_490684_object348:head# 2024-09-15T05:52:10.515 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.558 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a72003a8:::benchmark_data_smithi138_490684_object212:head# 2024-09-15T05:52:10.558 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.615 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a748c773:::benchmark_data_smithi138_490684_object50:head# 2024-09-15T05:52:10.615 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.674 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a75235d1:::benchmark_data_smithi138_490684_object288:head# 2024-09-15T05:52:10.674 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.724 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a79fbd7e:::benchmark_data_smithi138_490684_object254:head# 2024-09-15T05:52:10.724 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.758 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a7a1198d:::benchmark_data_smithi138_490684_object128:head# 2024-09-15T05:52:10.758 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.791 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a7fc3f4e:::benchmark_data_smithi138_490684_object292:head# 2024-09-15T05:52:10.791 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.825 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a815fbd0:::benchmark_data_smithi138_490684_object349:head# 2024-09-15T05:52:10.825 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.858 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a829aa2a:::benchmark_data_smithi138_490684_object27:head# 2024-09-15T05:52:10.859 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.900 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a82ed670:::benchmark_data_smithi138_490684_object190:head# 2024-09-15T05:52:10.900 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.959 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:a9eab397:::benchmark_data_smithi138_490684_object347:head# 2024-09-15T05:52:10.959 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:10.992 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aa72d9bf:::benchmark_data_smithi138_490684_object131:head# 2024-09-15T05:52:10.992 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.059 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aa736f85:::benchmark_data_smithi138_490684_object210:head# 2024-09-15T05:52:11.059 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.101 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aae00d6a:::benchmark_data_smithi138_490684_object424:head# 2024-09-15T05:52:11.101 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.151 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ab691369:::benchmark_data_smithi138_490684_object171:head# 2024-09-15T05:52:11.151 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.176 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ac0f7c3f:::benchmark_data_smithi138_490684_object38:head# 2024-09-15T05:52:11.176 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.201 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:acae68aa:::benchmark_data_smithi138_490684_object373:head# 2024-09-15T05:52:11.202 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.227 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:acd3dc4e:::benchmark_data_smithi138_490684_object343:head# 2024-09-15T05:52:11.227 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.251 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ad7d3fda:::benchmark_data_smithi138_490684_object276:head# 2024-09-15T05:52:11.252 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.277 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ada7bb33:::benchmark_data_smithi138_490684_object323:head# 2024-09-15T05:52:11.277 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.302 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ae507d31:::benchmark_data_smithi138_490684_object142:head# 2024-09-15T05:52:11.302 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.327 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:aed39694:::benchmark_data_smithi138_490684_object360:head# 2024-09-15T05:52:11.327 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.352 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:afc9dbd4:::benchmark_data_smithi138_490684_object398:head# 2024-09-15T05:52:11.353 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.378 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:afd27c18:::benchmark_data_smithi138_490684_object139:head# 2024-09-15T05:52:11.378 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b188dc72:::benchmark_data_smithi138_490684_object245:head# 2024-09-15T05:52:11.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.454 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b19c4782:::benchmark_data_smithi138_490684_object203:head# 2024-09-15T05:52:11.454 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.487 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b1fc5c75:::benchmark_data_smithi138_490684_object174:head# 2024-09-15T05:52:11.487 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.520 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b34319c5:::benchmark_data_smithi138_490684_object427:head# 2024-09-15T05:52:11.520 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.554 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b4506c06:::benchmark_data_smithi138_490684_object40:head# 2024-09-15T05:52:11.554 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.587 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b502b170:::benchmark_data_smithi138_490684_object227:head# 2024-09-15T05:52:11.587 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b5979a85:::benchmark_data_smithi138_490684_object388:head# 2024-09-15T05:52:11.620 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.654 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b602263f:::benchmark_data_smithi138_490684_object300:head# 2024-09-15T05:52:11.654 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.687 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b6ace319:::benchmark_data_smithi138_490684_object53:head# 2024-09-15T05:52:11.687 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.721 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b712434d:::benchmark_data_smithi138_490684_object37:head# 2024-09-15T05:52:11.721 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.754 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b76c65e1:::benchmark_data_smithi138_490684_object325:head# 2024-09-15T05:52:11.754 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.787 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8033cc1:::benchmark_data_smithi138_490684_object76:head# 2024-09-15T05:52:11.788 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.821 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b81dbbdb:::benchmark_data_smithi138_490684_object318:head# 2024-09-15T05:52:11.821 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.855 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-09-15T05:52:11.855 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.888 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8bfddf1:::benchmark_data_smithi138_490684_object298:head# 2024-09-15T05:52:11.888 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.922 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:b8c141d6:::benchmark_data_smithi138_490684_object247:head# 2024-09-15T05:52:11.922 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:11.955 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ba027fe8:::benchmark_data_smithi138_490684_object314:head# 2024-09-15T05:52:11.955 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.007 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ba6b497c:::benchmark_data_smithi138_490684_object413:head# 2024-09-15T05:52:12.008 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.080 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ba974ce9:::benchmark_data_smithi138_490684_object251:head# 2024-09-15T05:52:12.081 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.130 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:baac4be2:::benchmark_data_smithi138_490684_object209:head# 2024-09-15T05:52:12.131 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.164 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:babd7411:::benchmark_data_smithi138_490684_object30:head# 2024-09-15T05:52:12.164 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.198 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bc183f22:::benchmark_data_smithi138_490684_object382:head# 2024-09-15T05:52:12.198 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.231 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bce033f2:::benchmark_data_smithi138_490684_object175:head# 2024-09-15T05:52:12.231 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.265 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bdddf4cd:::benchmark_data_smithi138_490684_object293:head# 2024-09-15T05:52:12.265 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.298 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:becb43cd:::benchmark_data_smithi138_490684_object267:head# 2024-09-15T05:52:12.298 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.340 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:bee75cac:::benchmark_data_smithi138_490684_object44:head# 2024-09-15T05:52:12.340 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.373 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c126efe4:::benchmark_data_smithi138_490684_object23:head# 2024-09-15T05:52:12.373 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.407 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c18b114d:::benchmark_data_smithi138_490684_object417:head# 2024-09-15T05:52:12.407 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.440 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c255df2d:::benchmark_data_smithi138_490684_object370:head# 2024-09-15T05:52:12.440 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.474 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c3cdf8b8:::benchmark_data_smithi138_490684_object302:head# 2024-09-15T05:52:12.474 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c41f5dd1:::benchmark_data_smithi138_490684_object20:head# 2024-09-15T05:52:12.507 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.561 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c48d081e:::benchmark_data_smithi138_490684_object116:head# 2024-09-15T05:52:12.561 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.633 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c4da0bf7:::benchmark_data_smithi138_490684_object146:head# 2024-09-15T05:52:12.633 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.683 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c587d3eb:::benchmark_data_smithi138_490684_object234:head# 2024-09-15T05:52:12.683 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.733 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c5affd17:::benchmark_data_smithi138_490684_object346:head# 2024-09-15T05:52:12.733 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.766 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c710ddf5:::benchmark_data_smithi138_490684_object235:head# 2024-09-15T05:52:12.766 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.800 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7abfafd:::benchmark_data_smithi138_490684_object69:head# 2024-09-15T05:52:12.800 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.833 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c7ec31b7:::benchmark_data_smithi138_490684_object193:head# 2024-09-15T05:52:12.834 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.867 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c859c77d:::benchmark_data_smithi138_490684_object297:head# 2024-09-15T05:52:12.867 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.900 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c871f5cb:::benchmark_data_smithi138_490684_object395:head# 2024-09-15T05:52:12.901 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.934 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8986227:::benchmark_data_smithi138_490684_object24:head# 2024-09-15T05:52:12.934 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:12.967 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c8ad720c:::benchmark_data_smithi138_490684_object150:head# 2024-09-15T05:52:12.967 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.001 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c9e075cc:::benchmark_data_smithi138_490684_object226:head# 2024-09-15T05:52:13.001 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.034 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:c9f27f6b:::benchmark_data_smithi138_490684_object207:head# 2024-09-15T05:52:13.035 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.068 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ca14c54c:::benchmark_data_smithi138_490684_object31:head# 2024-09-15T05:52:13.068 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.101 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ca74b47b:::benchmark_data_smithi138_490684_object381:head# 2024-09-15T05:52:13.101 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.135 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ca8ea2fa:::benchmark_data_smithi138_490684_object406:head# 2024-09-15T05:52:13.135 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.168 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cabfa736:::benchmark_data_smithi138_490684_object273:head# 2024-09-15T05:52:13.168 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.202 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cb047965:::benchmark_data_smithi138_490684_object401:head# 2024-09-15T05:52:13.202 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cbddb5da:::benchmark_data_smithi138_490684_object315:head# 2024-09-15T05:52:13.235 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.269 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cc6702b8:::benchmark_data_smithi138_490684_object265:head# 2024-09-15T05:52:13.269 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.302 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cc7bc92b:::benchmark_data_smithi138_490684_object217:head# 2024-09-15T05:52:13.302 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.336 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ccbad358:::benchmark_data_smithi138_490684_object262:head# 2024-09-15T05:52:13.336 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.369 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ccfaaa43:::benchmark_data_smithi138_490684_object269:head# 2024-09-15T05:52:13.369 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd0d6cbc:::benchmark_data_smithi138_490684_object25:head# 2024-09-15T05:52:13.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.436 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cd49f35b:::benchmark_data_smithi138_490684_object168:head# 2024-09-15T05:52:13.436 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.470 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:cf060d03:::benchmark_data_smithi138_490684_object354:head# 2024-09-15T05:52:13.470 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.495 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d11b26be:::benchmark_data_smithi138_490684_object68:head# 2024-09-15T05:52:13.495 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.520 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d311a17c:::benchmark_data_smithi138_490684_object154:head# 2024-09-15T05:52:13.520 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.553 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d317ea78:::benchmark_data_smithi138_490684_object19:head# 2024-09-15T05:52:13.553 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.579 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d45838c9:::benchmark_data_smithi138_490684_object295:head# 2024-09-15T05:52:13.579 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.612 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d561ea00:::benchmark_data_smithi138_490684_object440:head# 2024-09-15T05:52:13.612 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.645 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d5a23c35:::benchmark_data_smithi138_490684_object216:head# 2024-09-15T05:52:13.646 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.679 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6d0b370:::benchmark_data_smithi138_490684_object75:head# 2024-09-15T05:52:13.679 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.712 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d6fd5d86:::benchmark_data_smithi138_490684_object66:head# 2024-09-15T05:52:13.712 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.746 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d7af5050:::benchmark_data_smithi138_490684_object153:head# 2024-09-15T05:52:13.746 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.779 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d878ba6f:::benchmark_data_smithi138_490684_object329:head# 2024-09-15T05:52:13.779 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.813 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8896898:::benchmark_data_smithi138_490684_object378:head# 2024-09-15T05:52:13.813 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.846 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d8ac82bf:::benchmark_data_smithi138_490684_object137:head# 2024-09-15T05:52:13.846 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.880 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d968376a:::benchmark_data_smithi138_490684_object441:head# 2024-09-15T05:52:13.880 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.913 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d9e0c3be:::benchmark_data_smithi138_490684_object320:head# 2024-09-15T05:52:13.913 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.948 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:d9fb1b48:::benchmark_data_smithi138_490684_object164:head# 2024-09-15T05:52:13.948 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:13.990 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:da4af231:::benchmark_data_smithi138_490684_object311:head# 2024-09-15T05:52:13.991 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.047 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:db2c59d2:::benchmark_data_smithi138_490684_object64:head# 2024-09-15T05:52:14.047 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.097 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:db3623e7:::benchmark_data_smithi138_490684_object0:head# 2024-09-15T05:52:14.097 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.131 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dbeebfbc:::benchmark_data_smithi138_490684_object425:head# 2024-09-15T05:52:14.131 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.164 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dccccbd3:::benchmark_data_smithi138_490684_object211:head# 2024-09-15T05:52:14.164 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.197 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dcf29248:::benchmark_data_smithi138_490684_object73:head# 2024-09-15T05:52:14.198 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.231 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd3cc6e6:::benchmark_data_smithi138_490684_object96:head# 2024-09-15T05:52:14.231 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.265 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd40e500:::benchmark_data_smithi138_490684_object39:head# 2024-09-15T05:52:14.265 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.298 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dd6a1e71:::benchmark_data_smithi138_490684_object279:head# 2024-09-15T05:52:14.298 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.331 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:de6a0aaa:::benchmark_data_smithi138_490684_object191:head# 2024-09-15T05:52:14.331 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.365 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:dea48036:::benchmark_data_smithi138_490684_object389:head# 2024-09-15T05:52:14.365 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.398 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:def79532:::benchmark_data_smithi138_490684_object308:head# 2024-09-15T05:52:14.398 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.432 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:df04885b:::benchmark_data_smithi138_490684_object99:head# 2024-09-15T05:52:14.432 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.465 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e07452fd:::benchmark_data_smithi138_490684_object335:head# 2024-09-15T05:52:14.465 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.499 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e0b05790:::benchmark_data_smithi138_490684_object85:head# 2024-09-15T05:52:14.499 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.532 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e0b656aa:::benchmark_data_smithi138_490684_object181:head# 2024-09-15T05:52:14.532 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.599 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e0e266eb:::benchmark_data_smithi138_490684_object112:head# 2024-09-15T05:52:14.599 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.666 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e106707e:::benchmark_data_smithi138_490684_object80:head# 2024-09-15T05:52:14.666 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.733 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e11daeb6:::benchmark_data_smithi138_490684_object341:head# 2024-09-15T05:52:14.733 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.767 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e14e303c:::benchmark_data_smithi138_490684_object366:head# 2024-09-15T05:52:14.767 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.800 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e2835e13:::benchmark_data_smithi138_490684_object71:head# 2024-09-15T05:52:14.800 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.834 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e2cb3a9c:::benchmark_data_smithi138_490684_object422:head# 2024-09-15T05:52:14.834 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.867 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e330ca21:::benchmark_data_smithi138_490684_object353:head# 2024-09-15T05:52:14.867 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.901 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e3827b88:::benchmark_data_smithi138_490684_object162:head# 2024-09-15T05:52:14.901 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.934 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e38bdbde:::benchmark_data_smithi138_490684_object429:head# 2024-09-15T05:52:14.934 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:14.967 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e39f10b6:::benchmark_data_smithi138_490684_object183:head# 2024-09-15T05:52:14.968 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.001 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e44b1845:::benchmark_data_smithi138_490684_object134:head# 2024-09-15T05:52:15.001 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.034 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e479c778:::benchmark_data_smithi138_490684_object365:head# 2024-09-15T05:52:15.035 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.068 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e4cd3c89:::benchmark_data_smithi138_490684_object144:head# 2024-09-15T05:52:15.068 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.101 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e604edd6:::benchmark_data_smithi138_490684_object32:head# 2024-09-15T05:52:15.101 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.135 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e658c1c4:::benchmark_data_smithi138_490684_object155:head# 2024-09-15T05:52:15.135 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.168 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e6881baa:::benchmark_data_smithi138_490684_object411:head# 2024-09-15T05:52:15.168 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.202 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e77e3c6d:::benchmark_data_smithi138_490684_object248:head# 2024-09-15T05:52:15.202 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.235 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e8354136:::benchmark_data_smithi138_490684_object376:head# 2024-09-15T05:52:15.235 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.269 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:e84b38d8:::benchmark_data_smithi138_490684_object26:head# 2024-09-15T05:52:15.269 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.302 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ea2a9c37:::benchmark_data_smithi138_490684_object444:head# 2024-09-15T05:52:15.302 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.336 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ea3a9c93:::benchmark_data_smithi138_490684_object330:head# 2024-09-15T05:52:15.336 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.378 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ea746610:::benchmark_data_smithi138_490684_object246:head# 2024-09-15T05:52:15.378 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.411 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:eaa02ef4:::benchmark_data_smithi138_490684_object72:head# 2024-09-15T05:52:15.411 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.445 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec164b11:::benchmark_data_smithi138_490684_object326:head# 2024-09-15T05:52:15.445 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.478 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec22600e:::benchmark_data_smithi138_490684_object294:head# 2024-09-15T05:52:15.478 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.512 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ec79d9e0:::benchmark_data_smithi138_490684_object108:head# 2024-09-15T05:52:15.512 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.545 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ecb3b783:::benchmark_data_smithi138_490684_object14:head# 2024-09-15T05:52:15.545 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.578 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ece76228:::benchmark_data_smithi138_490684_object313:head# 2024-09-15T05:52:15.578 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.612 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed25df26:::benchmark_data_smithi138_490684_object33:head# 2024-09-15T05:52:15.612 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.645 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ed850b4b:::benchmark_data_smithi138_490684_object428:head# 2024-09-15T05:52:15.645 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.670 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:edefdf02:::benchmark_data_smithi138_490684_object351:head# 2024-09-15T05:52:15.670 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.696 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ee044cee:::benchmark_data_smithi138_490684_object380:head# 2024-09-15T05:52:15.696 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.729 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ee80fec1:::benchmark_data_smithi138_490684_object202:head# 2024-09-15T05:52:15.729 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.754 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ef5377a3:::benchmark_data_smithi138_490684_object109:head# 2024-09-15T05:52:15.754 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.779 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:efac843c:::benchmark_data_smithi138_490684_object141:head# 2024-09-15T05:52:15.779 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.805 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:efc7cb10:::benchmark_data_smithi138_490684_object289:head# 2024-09-15T05:52:15.805 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.830 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:efe3dbc5:::benchmark_data_smithi138_490684_object132:head# 2024-09-15T05:52:15.830 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.854 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:efefe5df:::benchmark_data_smithi138_490684_object334:head# 2024-09-15T05:52:15.855 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.880 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f0198718:::benchmark_data_smithi138_490684_object170:head# 2024-09-15T05:52:15.880 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.913 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f0d693a5:::benchmark_data_smithi138_490684_object122:head# 2024-09-15T05:52:15.914 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.947 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f1880b91:::benchmark_data_smithi138_490684_object120:head# 2024-09-15T05:52:15.947 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:15.990 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f25e60ec:::benchmark_data_smithi138_490684_object359:head# 2024-09-15T05:52:15.990 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.031 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f2dc95dc:::benchmark_data_smithi138_490684_object229:head# 2024-09-15T05:52:16.031 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.064 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f347b8ae:::benchmark_data_smithi138_490684_object54:head# 2024-09-15T05:52:16.064 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.098 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f3c9e336:::benchmark_data_smithi138_490684_object283:head# 2024-09-15T05:52:16.098 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.131 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f3db25d5:::benchmark_data_smithi138_490684_object438:head# 2024-09-15T05:52:16.131 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.173 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f4244c3a:::benchmark_data_smithi138_490684_object208:head# 2024-09-15T05:52:16.173 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.206 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f55f7de3:::benchmark_data_smithi138_490684_object405:head# 2024-09-15T05:52:16.207 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.240 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f5bb8d1e:::benchmark_data_smithi138_490684_object123:head# 2024-09-15T05:52:16.240 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.273 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f619712b:::benchmark_data_smithi138_490684_object409:head# 2024-09-15T05:52:16.273 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.307 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f65a5eaa:::benchmark_data_smithi138_490684_object436:head# 2024-09-15T05:52:16.307 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.340 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f65f6043:::benchmark_data_smithi138_490684_object275:head# 2024-09-15T05:52:16.340 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.373 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f7c6c358:::benchmark_data_smithi138_490684_object337:head# 2024-09-15T05:52:16.374 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.407 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f8e1def2:::benchmark_data_smithi138_490684_object196:head# 2024-09-15T05:52:16.407 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.472 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f8f42a80:::benchmark_data_smithi138_490684_object446:head# 2024-09-15T05:52:16.472 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.507 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:f93bea36:::benchmark_data_smithi138_490684_object377:head# 2024-09-15T05:52:16.507 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.541 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fa473e4c:::benchmark_data_smithi138_490684_object125:head# 2024-09-15T05:52:16.541 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.616 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fb9decce:::benchmark_data_smithi138_490684_object396:head# 2024-09-15T05:52:16.616 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.666 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fbc6b792:::benchmark_data_smithi138_490684_object177:head# 2024-09-15T05:52:16.666 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.717 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fe5d71ab:::benchmark_data_smithi138_490684_object107:head# 2024-09-15T05:52:16.717 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.750 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:feb2f8f3:::benchmark_data_smithi138_490684_object421:head# 2024-09-15T05:52:16.750 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.784 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:fed63b8b:::benchmark_data_smithi138_490684_object90:head# 2024-09-15T05:52:16.784 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.817 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ff3c1513:::benchmark_data_smithi138_490684_object57:head# 2024-09-15T05:52:16.817 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.851 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ff75089e:::benchmark_data_smithi138_490684_object79:head# 2024-09-15T05:52:16.851 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.884 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffaa5f03:::benchmark_data_smithi138_490684_object147:head# 2024-09-15T05:52:16.884 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.943 INFO:tasks.workunit.client.0.smithi138.stdout:Write #1:ffcc0e7c:::benchmark_data_smithi138_490684_object412:head# 2024-09-15T05:52:16.943 INFO:tasks.workunit.client.0.smithi138.stdout:snapset 0=[]:{} 2024-09-15T05:52:16.976 INFO:tasks.workunit.client.0.smithi138.stdout:write_pg epoch 9 info 1.0( v 9'449 (0'0,9'449] local-lis/les=7/8 n=449 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-09-15T05:52:17.010 INFO:tasks.workunit.client.0.smithi138.stdout:Import successful 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-09-15T05:52:17.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:17.530 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:52:17.531 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-09-15T05:52:17.532 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=5b49d27b-2788-4342-b97c-2c4c58679608 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.466970/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:52:17.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:52:17.556 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:17.555+0000 7f37ecd49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:17.558 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:17.558+0000 7f37ecd49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:17.560 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:17.559+0000 7f37ecd49440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:17.932 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:52:17.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:52:17.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:17.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:18.074 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:18.073+0000 7f37ecd49440 -1 Falling back to public interface 2024-09-15T05:52:18.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:19.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:19.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:19.284 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:52:19.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:52:19.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:19.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:19.637 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:20.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:20.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:20.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:52:20.639 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:52:20.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:20.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:21.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:21.139 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:21.138+0000 7f37ecd49440 -1 osd.0 18 log_to_monitors true 2024-09-15T05:52:22.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:22.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:22.012 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:52:22.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:52:22.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:22.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:22.454 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/493497,v1:127.0.0.1:6803/493497] [v2:127.0.0.1:6804/493497,v1:127.0.0.1:6805/493497] exists,up 32ad5dc5-562e-406a-beed-5a5ad5ffb272 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:52:22.456 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:52:22.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:52:22.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:52:22.457 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:52:22.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:52:22.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:52:22.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:52:22.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:52:22.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:52:22.542 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:52:22.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids=0 2024-09-15T05:52:22.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:52:22.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:52:22.889 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:52:22.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=90194313219 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 90194313219 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-90194313219' 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-90194313219 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:52:22.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:52:22.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-90194313219 2024-09-15T05:52:22.997 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:52:22.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=90194313219 2024-09-15T05:52:22.998 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 90194313219 2024-09-15T05:52:22.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-09-15T05:52:22.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:52:23.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 1 -lt 90194313219 2024-09-15T05:52:23.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:52:24.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:52:24.347 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:52:24.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 90194313219 -lt 90194313219 2024-09-15T05:52:24.741 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:52:24.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:52:24.742 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:52:25.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-09-15T05:52:25.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:52:25.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:52:25.211 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:52:25.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:52:25.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:52:25.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:52:25.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:52:25.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-09-15T05:52:25.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:52:25.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:52:25.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:52:26.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:52:26.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:52:26.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:52:26.150 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:52:26.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:52:26.151 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:52:26.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:52:26.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:52:26.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:52:26.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:52:26.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:52:26.154 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:52:26.155 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:52:26.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:52:26.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:52:26.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.165 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:52:26.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/pg-split-merge 0 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T05:52:26.167 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:52:26.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:52:26.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:52:26.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:52:26.169 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:52:26.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:52:26.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:52:26.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:52:26.171 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:52:26.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:52:26.172 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:52:26.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T05:52:26.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-09-15T05:52:26.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:52:26.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.466970 2024-09-15T05:52:26.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.466970 2024-09-15T05:52:26.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:52:26.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:52:26.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T05:52:26.177 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T05:52:26.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T05:52:26.242 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-09-15T05:52:26.242 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-09-15T05:52:26.305 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T05:52:26.310 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T05:52:26.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:52:26.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/repeer-on-acting-back 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-09-15T05:52:26.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-09-15T05:52:26.312 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none ' 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T05:52:26.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:52:26.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:52:26.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:52:26.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:52:26.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:52:26.317 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:52:26.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:52:26.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:52:26.318 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:52:26.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:52:26.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:52:26.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-15T05:52:26.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:52:26.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.321 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:26.321 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494363 2024-09-15T05:52:26.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:52:26.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:52:26.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-09-15T05:52:26.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:52:26.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.323 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:26.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.494363 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-09-15T05:52:26.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-09-15T05:52:26.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-09-15T05:52:26.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-09-15T05:52:26.326 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-09-15T05:52:26.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:52:26.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-09-15T05:52:26.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-09-15T05:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:52:26.665 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:52:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:52:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:26.666 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:52:26.666 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:52:26.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:52:26.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:52:26.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:52:26.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:52:26.928 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494363/ceph-mon.a.asok 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:52:26.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494363/ceph-mon.a.asok config get fsid 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:52:27.000 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:52:27.001 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:27.001 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:27.001 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494363/ceph-mon.a.asok 2024-09-15T05:52:27.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:52:27.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494363/ceph-mon.a.asok config get mon_host 2024-09-15T05:52:27.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-09-15T05:52:27.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-09-15T05:52:27.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:52:27.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:52:27.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:52:27.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-09-15T05:52:27.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:52:27.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:52:27.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:52:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:52:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-09-15T05:52:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:52:27.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:52:27.950 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:52:27.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=db4bfdb8-3bc6-41a0-826c-203add7c41ab 2024-09-15T05:52:27.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 db4bfdb8-3bc6-41a0-826c-203add7c41ab' 2024-09-15T05:52:27.953 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 db4bfdb8-3bc6-41a0-826c-203add7c41ab 2024-09-15T05:52:27.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:52:27.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAbduZmSTGpORAAAWfZRujGicfe6KZcdcQkjg== 2024-09-15T05:52:27.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAbduZmSTGpORAAAWfZRujGicfe6KZcdcQkjg=="}' 2024-09-15T05:52:27.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new db4bfdb8-3bc6-41a0-826c-203add7c41ab -i td/repeer-on-acting-back/0/new.json 2024-09-15T05:52:28.288 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:52:28.298 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-09-15T05:52:28.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAbduZmSTGpORAAAWfZRujGicfe6KZcdcQkjg== --osd-uuid db4bfdb8-3bc6-41a0-826c-203add7c41ab 2024-09-15T05:52:28.329 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.328+0000 7f1e35935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:28.334 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.334+0000 7f1e35935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:28.335 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.335+0000 7f1e35935440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:28.336 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.335+0000 7f1e35935440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-15T05:52:28.336 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.335+0000 7f1e35935440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-15T05:52:28.336 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.335+0000 7f1e35935440 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-09-15T05:52:28.375 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:28.374+0000 7f1e35935440 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-09-15T05:52:33.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-09-15T05:52:33.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:52:33.677 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:52:33.677 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:52:33.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:52:34.168 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:52:34.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:52:34.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:52:34.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:52:34.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:52:34.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:52:34.197 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:34.196+0000 7fe3d6321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:34.223 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:34.223+0000 7fe3d6321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:34.225 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:34.224+0000 7fe3d6321440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:52:34.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:34.613 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:52:34.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:52:34.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:34.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:34.765 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:34.764+0000 7fe3d6321440 -1 Falling back to public interface 2024-09-15T05:52:34.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:35.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:35.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:35.969 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:52:35.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:52:35.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:35.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:36.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:37.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:37.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:37.329 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:52:37.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:52:37.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:37.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:37.344 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:52:37.343+0000 7fe3d6321440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:52:37.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:38.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:38.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:38.722 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:52:38.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:52:38.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:38.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:39.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:40.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:40.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:40.183 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:52:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:52:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:40.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:40.579 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:41.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:41.581 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:52:41.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:52:41.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:41.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:42.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:43.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:43.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:43.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:52:43.030 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:52:43.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:43.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:43.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:44.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:44.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:44.418 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:52:44.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:52:44.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:44.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:44.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:45.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:45.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:45.783 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:52:45.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:52:45.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:45.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:46.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:52:47.180 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:52:47.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:47.181 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:47.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:48.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:48.576 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:52:48.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:52:48.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:48.577 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:48.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:49.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:49.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:49.988 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:52:49.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:52:49.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:49.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:50.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:51.413 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:52:51.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:52:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:51.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:52.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:53.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:53.034 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:52:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:52:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:53.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:53.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:54.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:54.551 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:54.551 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:52:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:52:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:54.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:55.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:56.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:56.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:56.272 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:52:56.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:52:56.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:56.273 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:57.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:58.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:58.311 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:52:58.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:58.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:52:58.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:58.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:52:58.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:52:59.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:52:59.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:52:59.815 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:52:59.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:52:59.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:52:59.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:53:00.207 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/495457,v1:127.0.0.1:6803/495457] [v2:127.0.0.1:6804/495457,v1:127.0.0.1:6805/495457] exists,up db4bfdb8-3bc6-41a0-826c-203add7c41ab 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:53:00.211 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:53:00.212 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:53:00.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:53:00.214 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 92617913-c8db-4ce4-b082-6c0e7ae6a92c 2024-09-15T05:53:00.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=92617913-c8db-4ce4-b082-6c0e7ae6a92c 2024-09-15T05:53:00.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 92617913-c8db-4ce4-b082-6c0e7ae6a92c' 2024-09-15T05:53:00.215 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:53:00.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA8duZmh+6bDRAAC6NU2DwpI66aw3lESuU7RA== 2024-09-15T05:53:00.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA8duZmh+6bDRAAC6NU2DwpI66aw3lESuU7RA=="}' 2024-09-15T05:53:00.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 92617913-c8db-4ce4-b082-6c0e7ae6a92c -i td/repeer-on-acting-back/1/new.json 2024-09-15T05:53:00.710 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:53:00.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-09-15T05:53:00.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8duZmh+6bDRAAC6NU2DwpI66aw3lESuU7RA== --osd-uuid 92617913-c8db-4ce4-b082-6c0e7ae6a92c 2024-09-15T05:53:00.751 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.751+0000 7f18785fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:00.753 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.752+0000 7f18785fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:00.754 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.753+0000 7f18785fe440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:00.755 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.754+0000 7f18785fe440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-15T05:53:00.755 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.754+0000 7f18785fe440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-15T05:53:00.755 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.754+0000 7f18785fe440 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-09-15T05:53:00.800 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:00.799+0000 7f18785fe440 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-09-15T05:53:06.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-09-15T05:53:06.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:53:06.416 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:53:06.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:53:06.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:53:06.908 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:53:06.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:53:06.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:53:06.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:53:06.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:53:06.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:53:06.936 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:06.935+0000 7f52fdc15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:06.938 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:06.937+0000 7f52fdc15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:06.939 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:06.938+0000 7f52fdc15440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:07.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:53:07.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:53:07.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:53:07.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:07.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:07.458 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:07.457+0000 7f52fdc15440 -1 Falling back to public interface 2024-09-15T05:53:07.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:08.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:08.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:08.679 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:53:08.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:53:08.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:08.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:09.040 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:09.962 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:09.961+0000 7f52fdc15440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:53:10.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:10.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:10.041 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:53:10.042 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:53:10.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:10.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:10.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:11.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:11.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:11.449 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:53:11.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:53:11.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:11.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:11.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:12.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:12.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:12.860 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:53:12.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:53:12.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:12.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:13.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:14.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:14.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:14.262 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:53:14.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:53:14.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:14.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:14.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:15.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:15.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:15.746 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:53:15.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:53:15.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:15.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:16.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:17.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:17.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:17.310 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:53:17.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:53:17.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:17.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:17.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:18.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:18.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:18.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:53:18.676 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:53:18.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:18.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:19.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:20.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:20.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:20.201 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:53:20.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:53:20.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:20.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:20.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:21.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:21.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:21.568 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:53:21.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:53:21.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:21.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:22.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:23.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:23.079 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:53:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:53:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:23.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:23.699 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:24.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:24.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:24.701 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:53:24.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:53:24.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:24.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:25.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:26.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:26.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:26.567 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:53:26.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:53:26.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:26.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:27.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:28.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:28.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:28.370 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:53:28.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:53:28.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:28.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:29.163 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:30.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:30.164 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:30.164 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:53:30.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:53:30.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:30.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:32.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:32.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:32.336 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:53:32.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:53:32.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:32.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:32.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:33.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:33.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:33.763 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:53:33.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:53:33.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:33.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:53:34.169 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/497214,v1:127.0.0.1:6811/497214] [v2:127.0.0.1:6812/497214,v1:127.0.0.1:6813/497214] exists,up 92617913-c8db-4ce4-b082-6c0e7ae6a92c 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:53:34.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:53:34.174 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:53:34.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:53:34.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 223de97b-974f-44df-a396-2a12c0294c86' 2024-09-15T05:53:34.177 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:53:34.177 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:53:34.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeduZmjy5uCxAAxA90skwISEPxxsb5Ol6RbQ== 2024-09-15T05:53:34.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeduZmjy5uCxAAxA90skwISEPxxsb5Ol6RbQ=="}' 2024-09-15T05:53:34.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 223de97b-974f-44df-a396-2a12c0294c86 -i td/repeer-on-acting-back/2/new.json 2024-09-15T05:53:34.631 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:53:34.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-09-15T05:53:34.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeduZmjy5uCxAAxA90skwISEPxxsb5Ol6RbQ== --osd-uuid 223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:53:34.673 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.672+0000 7f636e97b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:34.676 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.674+0000 7f636e97b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:34.676 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.675+0000 7f636e97b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:34.676 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.675+0000 7f636e97b440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-15T05:53:34.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.675+0000 7f636e97b440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-15T05:53:34.677 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.675+0000 7f636e97b440 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-09-15T05:53:34.722 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:34.721+0000 7f636e97b440 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-09-15T05:53:40.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-09-15T05:53:40.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:53:40.486 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:53:40.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:53:40.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:53:40.975 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:53:40.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:53:40.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:53:40.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:53:40.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:53:40.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:53:41.004 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:41.003+0000 7f64f8e11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:41.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:41.004+0000 7f64f8e11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:41.007 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:41.005+0000 7f64f8e11440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:53:41.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:41.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:53:41.360 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:53:41.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:41.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:41.523 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:41.522+0000 7f64f8e11440 -1 Falling back to public interface 2024-09-15T05:53:41.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:42.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:42.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:42.737 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:53:42.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:53:42.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:42.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:43.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:44.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:44.102 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:44.102 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:53:44.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:53:44.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:44.103 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:44.133 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:53:44.132+0000 7f64f8e11440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:53:44.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:45.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:45.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:45.493 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:53:45.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:53:45.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:45.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:45.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:46.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:46.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:46.997 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:53:46.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:53:46.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:46.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:47.377 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:48.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:48.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:48.379 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:53:48.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:53:48.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:48.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:48.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:49.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:49.760 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:53:49.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:49.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:53:49.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:49.761 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:50.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:51.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:51.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:51.297 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:53:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:53:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:51.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:51.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:52.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:52.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:52.714 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:53:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:53:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:52.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:53.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:54.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:54.186 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:53:54.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:54.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:53:54.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:54.187 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:54.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:55.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:55.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:55.634 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:53:55.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:53:55.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:55.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:56.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:57.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:57.033 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:53:57.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:57.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:53:57.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:57.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:57.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:53:58.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:53:58.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:53:58.809 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:53:58.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:53:58.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:53:58.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:53:59.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:00.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:00.684 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:00.685 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:54:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:54:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:00.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:54:01.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:02.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:02.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:02.455 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:54:02.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:54:02.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:02.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:54:03.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:04.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:04.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:04.135 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:54:04.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:54:04.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:04.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:54:05.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:06.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:06.366 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:54:06.366 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:06.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:54:06.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:06.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:54:06.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:07.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:07.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:07.983 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:54:07.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:54:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:07.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:54:08.373 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/498974,v1:127.0.0.1:6819/498974] [v2:127.0.0.1:6820/498974,v1:127.0.0.1:6821/498974] exists,up 223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:54:08.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:54:08.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:54:08.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:54:08.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:54:08.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-09-15T05:54:08.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:54:08.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-09-15T05:54:08.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:54:08.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:54:08.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:54:08.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-09-15T05:54:08.384 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:54:08.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b 2024-09-15T05:54:08.384 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b' 2024-09-15T05:54:08.384 INFO:tasks.workunit.client.0.smithi138.stdout:add osd3 6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b 2024-09-15T05:54:08.385 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:54:08.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAduZmrPx/FxAAeg8XPy9AfEOkeTkUe4Ng8A== 2024-09-15T05:54:08.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAduZmrPx/FxAAeg8XPy9AfEOkeTkUe4Ng8A=="}' 2024-09-15T05:54:08.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b -i td/repeer-on-acting-back/3/new.json 2024-09-15T05:54:08.861 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:54:08.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-09-15T05:54:08.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAduZmrPx/FxAAeg8XPy9AfEOkeTkUe4Ng8A== --osd-uuid 6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b 2024-09-15T05:54:08.903 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.902+0000 7f7680f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:08.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.929+0000 7f7680f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:08.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.930+0000 7f7680f5e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:08.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.930+0000 7f7680f5e440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-15T05:54:08.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.930+0000 7f7680f5e440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-15T05:54:08.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.930+0000 7f7680f5e440 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-09-15T05:54:08.961 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:08.960+0000 7f7680f5e440 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-09-15T05:54:14.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-09-15T05:54:14.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:54:14.489 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd3 key to auth repository 2024-09-15T05:54:14.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-15T05:54:14.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:54:14.985 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.3 2024-09-15T05:54:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-15T05:54:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:54:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:54:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:54:14.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:54:15.014 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:15.013+0000 7fd12e633440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:15.016 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:15.015+0000 7fd12e633440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:15.017 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:15.016+0000 7fd12e633440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:54:15.345 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:54:15.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:15.346 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:15.534 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:15.533+0000 7fd12e633440 -1 Falling back to public interface 2024-09-15T05:54:15.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:16.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:16.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:16.732 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:54:16.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:54:16.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:16.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:17.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:18.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:18.088+0000 7fd12e633440 -1 osd.3 0 log_to_monitors true 2024-09-15T05:54:18.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:18.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:18.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:54:18.091 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:54:18.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:18.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:18.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:19.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:19.575 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:19.575 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:54:19.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:54:19.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:19.576 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:19.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:20.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:20.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:20.998 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:54:20.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:54:20.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:20.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:21.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:22.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:22.387 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:22.388 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:54:22.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:54:22.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:22.389 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:22.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:23.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:23.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:23.862 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:54:23.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:54:23.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:23.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:24.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:25.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:25.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:25.292 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:54:25.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:54:25.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:25.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:25.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:26.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:26.713 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:54:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:54:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:26.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:27.193 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:28.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:28.194 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:28.195 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:54:28.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:54:28.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:28.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:28.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:29.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:29.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:29.560 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:54:29.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:54:29.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:29.560 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:30.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:31.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:31.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:31.399 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:54:31.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:54:31.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:31.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:32.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:33.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:33.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:33.052 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:54:33.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:54:33.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:33.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:33.801 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:34.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:34.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:34.802 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:54:34.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:54:34.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:34.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:35.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:36.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:36.505 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:36.506 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:54:36.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:54:36.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:36.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:37.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:38.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:38.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:38.219 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:54:38.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:54:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:38.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:39.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:40.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:40.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:40.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:54:40.160 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:54:40.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:40.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:40.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:41.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:41.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:41.613 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:54:41.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:54:41.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:41.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-15T05:54:41.979 INFO:tasks.workunit.client.0.smithi138.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/500729,v1:127.0.0.1:6827/500729] [v2:127.0.0.1:6828/500729,v1:127.0.0.1:6829/500729] exists,up 6dd3fdd8-6b75-4b9a-b8a0-64f672d0438b 2024-09-15T05:54:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:54:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:54:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:54:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:54:41.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:54:41.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:54:41.986 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:54:41.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-09-15T05:54:41.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:54:41.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=746a8b78-86b8-490f-a16c-7e04d1675b64 2024-09-15T05:54:41.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 746a8b78-86b8-490f-a16c-7e04d1675b64' 2024-09-15T05:54:41.988 INFO:tasks.workunit.client.0.smithi138.stdout:add osd4 746a8b78-86b8-490f-a16c-7e04d1675b64 2024-09-15T05:54:41.988 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:54:42.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiduZm8T8QABAAsUs9cmooaqeENpKYVU3bSA== 2024-09-15T05:54:42.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiduZm8T8QABAAsUs9cmooaqeENpKYVU3bSA=="}' 2024-09-15T05:54:42.002 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 746a8b78-86b8-490f-a16c-7e04d1675b64 -i td/repeer-on-acting-back/4/new.json 2024-09-15T05:54:42.495 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:54:42.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-09-15T05:54:42.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiduZm8T8QABAAsUs9cmooaqeENpKYVU3bSA== --osd-uuid 746a8b78-86b8-490f-a16c-7e04d1675b64 2024-09-15T05:54:42.538 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.537+0000 7f4ecff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:42.539 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.538+0000 7f4ecff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:42.540 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.539+0000 7f4ecff23440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:42.541 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.540+0000 7f4ecff23440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-15T05:54:42.541 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.540+0000 7f4ecff23440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-15T05:54:42.541 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.540+0000 7f4ecff23440 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-09-15T05:54:42.637 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:42.635+0000 7f4ecff23440 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-09-15T05:54:48.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-09-15T05:54:48.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:54:48.218 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd4 key to auth repository 2024-09-15T05:54:48.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-09-15T05:54:48.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:54:48.708 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.4 2024-09-15T05:54:48.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-09-15T05:54:48.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:54:48.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:54:48.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:54:48.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:54:48.736 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:48.735+0000 7f811720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:48.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:48.738+0000 7f811720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:48.739 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:48.739+0000 7f811720a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:54:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-09-15T05:54:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:54:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-09-15T05:54:49.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:54:49.066 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:54:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:54:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:54:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:49.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:49.280 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:49.279+0000 7f811720a440 -1 Falling back to public interface 2024-09-15T05:54:49.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:50.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:50.422 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:54:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:54:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:50.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:50.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:51.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:51.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:51.817 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:54:51.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:54:51.817 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:51.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:51.830 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:54:51.829+0000 7f811720a440 -1 osd.4 0 log_to_monitors true 2024-09-15T05:54:52.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:53.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:53.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:53.283 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:54:53.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:54:53.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:53.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:53.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:54.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:54.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:54.832 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:54:54.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:54:54.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:54.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:55.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:56.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:56.265 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:54:56.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:56.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:54:56.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:56.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:56.685 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:57.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:57.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:57.687 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:54:57.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:54:57.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:57.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:58.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:54:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:54:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:54:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:54:59.073 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:54:59.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:54:59.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:54:59.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:00.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:00.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:00.598 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:55:00.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:55:00.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:00.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:00.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:01.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:01.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:01.996 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:55:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:55:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:01.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:02.402 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:03.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:03.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:03.404 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:55:03.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:55:03.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:03.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:03.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:04.772 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:55:04.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:55:04.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:04.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:05.440 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:06.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:06.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:06.442 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:55:06.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:55:06.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:06.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:07.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:08.435 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:08.435 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:55:08.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:55:08.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:08.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:09.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:10.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:10.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:10.304 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:55:10.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:55:10.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:10.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:10.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:11.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:11.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:11.896 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:55:11.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:55:11.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:11.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:12.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:13.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:13.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:13.845 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:55:13.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:55:13.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:13.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:14.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:15.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:15.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:15.428 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:55:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:55:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:15.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-09-15T05:55:15.854 INFO:tasks.workunit.client.0.smithi138.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/502483,v1:127.0.0.1:6835/502483] [v2:127.0.0.1:6836/502483,v1:127.0.0.1:6837/502483] exists,up 746a8b78-86b8-490f-a16c-7e04d1675b64 2024-09-15T05:55:15.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:55:15.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:55:15.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:55:15.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:55:15.860 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:55:15.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2613d9f5-88f4-4f42-a5fc-28fcb423324a 2024-09-15T05:55:15.862 INFO:tasks.workunit.client.0.smithi138.stdout:add osd5 2613d9f5-88f4-4f42-a5fc-28fcb423324a 2024-09-15T05:55:15.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 2613d9f5-88f4-4f42-a5fc-28fcb423324a' 2024-09-15T05:55:15.863 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:55:15.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDDduZmrfAxNBAAS/ng+6S/BDONaI/A74TBOw== 2024-09-15T05:55:15.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDDduZmrfAxNBAAS/ng+6S/BDONaI/A74TBOw=="}' 2024-09-15T05:55:15.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2613d9f5-88f4-4f42-a5fc-28fcb423324a -i td/repeer-on-acting-back/5/new.json 2024-09-15T05:55:16.582 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:55:16.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-09-15T05:55:16.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDDduZmrfAxNBAAS/ng+6S/BDONaI/A74TBOw== --osd-uuid 2613d9f5-88f4-4f42-a5fc-28fcb423324a 2024-09-15T05:55:16.623 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.623+0000 7f6554413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:16.625 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.625+0000 7f6554413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:16.626 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.626+0000 7f6554413440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.626+0000 7f6554413440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-15T05:55:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.626+0000 7f6554413440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-15T05:55:16.627 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.626+0000 7f6554413440 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-09-15T05:55:16.673 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:16.672+0000 7f6554413440 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-09-15T05:55:22.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-09-15T05:55:22.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:55:22.485 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd5 key to auth repository 2024-09-15T05:55:22.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-09-15T05:55:22.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:55:22.975 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.5 2024-09-15T05:55:22.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-09-15T05:55:22.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:55:22.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:55:22.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:55:22.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:55:23.004 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:23.003+0000 7f94c3a30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:23.005 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:23.005+0000 7f94c3a30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:23.006 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:23.006+0000 7f94c3a30440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:55:23.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-09-15T05:55:23.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:55:23.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-09-15T05:55:23.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:55:23.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:55:23.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:23.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:55:23.376 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:55:23.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:23.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:23.528 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:23.527+0000 7f94c3a30440 -1 Falling back to public interface 2024-09-15T05:55:23.729 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:24.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:24.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:24.731 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:55:24.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:55:24.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:24.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:25.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:26.059 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:55:26.058+0000 7f94c3a30440 -1 osd.5 0 log_to_monitors true 2024-09-15T05:55:26.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:26.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:26.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:55:26.138 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:55:26.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:26.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:26.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:27.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:27.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:27.588 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:55:27.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:55:27.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:27.588 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:28.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:29.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:29.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:29.020 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:55:29.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:55:29.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:29.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:29.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:30.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:30.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:30.430 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:55:30.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:55:30.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:30.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:30.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:31.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:31.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:31.895 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:55:31.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:55:31.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:31.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:32.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:33.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:33.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:33.412 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:55:33.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:55:33.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:33.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:33.918 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:34.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:55:34.921 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:55:34.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:34.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:36.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:36.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:36.296 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:55:36.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:55:36.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:36.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:36.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:37.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:37.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:37.662 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:55:37.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:55:37.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:37.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:38.239 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:39.240 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:39.241 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:55:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:55:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:39.241 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:39.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:40.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:40.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:40.785 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:55:40.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:55:40.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:40.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:41.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:42.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:42.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:42.586 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:55:42.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:55:42.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:42.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:43.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:44.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:44.237 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:44.237 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:55:44.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:55:44.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:44.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:45.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:46.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:46.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:46.231 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:55:46.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:55:46.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:46.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:47.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:48.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:48.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:48.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:55:48.109 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:55:48.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:48.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:49.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:55:50.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:55:50.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:55:50.015 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:55:50.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:55:50.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:55:50.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-09-15T05:55:50.454 INFO:tasks.workunit.client.0.smithi138.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/504237,v1:127.0.0.1:6843/504237] [v2:127.0.0.1:6844/504237,v1:127.0.0.1:6845/504237] exists,up 2613d9f5-88f4-4f42-a5fc-28fcb423324a 2024-09-15T05:55:50.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:55:50.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:55:50.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:55:50.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-09-15T05:55:50.455 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-09-15T05:55:51.025 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:55:51.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-15T05:55:52.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-09-15T05:55:52.874 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 size to 3 2024-09-15T05:55:52.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-09-15T05:55:53.728 INFO:tasks.workunit.client.0.smithi138.stderr:set pool 1 min_size to 2 2024-09-15T05:55:53.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-09-15T05:55:53.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-09-15T05:55:53.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-09-15T05:55:54.073 INFO:tasks.workunit.client.0.smithi138.stderr:dumped pools 2024-09-15T05:55:54.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-09-15T05:55:54.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-09-15T05:55:54.084 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-09-15T05:55:54.867 INFO:tasks.workunit.client.0.smithi138.stderr:set require_min_compat_client to luminous 2024-09-15T05:55:54.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-09-15T05:55:55.697 INFO:tasks.workunit.client.0.smithi138.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-09-15T05:55:55.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-09-15T05:55:55.708 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:55:55.708 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:55:56.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.171 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:55:56.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836521 2024-09-15T05:55:56.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836521 2024-09-15T05:55:56.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521' 2024-09-15T05:55:56.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:55:56.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672993 2024-09-15T05:55:56.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672993 2024-09-15T05:55:56.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993' 2024-09-15T05:55:56.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.439 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509469 2024-09-15T05:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509469 2024-09-15T05:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509469' 2024-09-15T05:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.550 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:55:56.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345941 2024-09-15T05:55:56.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345941 2024-09-15T05:55:56.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509469 3-85899345941' 2024-09-15T05:55:56.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:55:56.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182417 2024-09-15T05:55:56.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182417 2024-09-15T05:55:56.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509469 3-85899345941 4-107374182417' 2024-09-15T05:55:56.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:55:56.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:55:56.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018887 2024-09-15T05:55:56.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018887 2024-09-15T05:55:56.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509469 3-85899345941 4-107374182417 5-128849018887' 2024-09-15T05:55:56.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:55:56.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836521 2024-09-15T05:55:56.884 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:55:56.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:55:56.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836521 2024-09-15T05:55:56.885 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:55:56.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836521 2024-09-15T05:55:56.886 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836521 2024-09-15T05:55:56.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-09-15T05:55:56.887 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:55:57.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836519 -lt 21474836521 2024-09-15T05:55:57.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:55:58.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:55:58.245 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:55:58.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836521 -lt 21474836521 2024-09-15T05:55:58.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:55:58.613 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672993 2024-09-15T05:55:58.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:55:58.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:55:58.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672993 2024-09-15T05:55:58.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:55:58.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672993 2024-09-15T05:55:58.616 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672993 2024-09-15T05:55:58.616 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-09-15T05:55:58.616 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:55:58.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672993 -lt 42949672993 2024-09-15T05:55:58.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:55:58.969 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509469 2024-09-15T05:55:58.970 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:55:58.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:55:58.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509469 2024-09-15T05:55:58.971 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:55:58.972 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509469 2024-09-15T05:55:58.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509469 2024-09-15T05:55:58.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-09-15T05:55:58.972 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:55:59.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509469 -lt 64424509469 2024-09-15T05:55:59.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:55:59.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345941 2024-09-15T05:55:59.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:55:59.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:55:59.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345941 2024-09-15T05:55:59.326 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:55:59.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345941 2024-09-15T05:55:59.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-09-15T05:55:59.328 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345941 2024-09-15T05:55:59.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:55:59.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345941 -lt 85899345941 2024-09-15T05:55:59.691 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:55:59.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182417 2024-09-15T05:55:59.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:55:59.692 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:55:59.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182417 2024-09-15T05:55:59.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:55:59.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182417 2024-09-15T05:55:59.694 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182417 2024-09-15T05:55:59.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182417' 2024-09-15T05:55:59.694 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:56:00.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182417 -lt 107374182417 2024-09-15T05:56:00.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:00.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018887 2024-09-15T05:56:00.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:00.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:56:00.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018887 2024-09-15T05:56:00.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:00.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018887 2024-09-15T05:56:00.073 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018887 2024-09-15T05:56:00.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018887' 2024-09-15T05:56:00.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018887 -lt 128849018887 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:56:00.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T05:56:00.431 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:56:00.431 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:56:00.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:56:00.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:56:00.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:56:00.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T05:56:00.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:56:00.515 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:56:00.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:56:00.516 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:56:00.516 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:56:00.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:56:00.909 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:56:00.909 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:56:00.909 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:56:00.909 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:56:00.910 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:56:00.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:56:00.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:00.910 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:56:01.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836524 2024-09-15T05:56:01.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836524 2024-09-15T05:56:01.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524' 2024-09-15T05:56:01.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:01.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:56:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949672996 2024-09-15T05:56:01.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949672996 2024-09-15T05:56:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-15T05:56:01.138 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:01.138 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:56:01.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509472 2024-09-15T05:56:01.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509472 2024-09-15T05:56:01.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509472' 2024-09-15T05:56:01.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:01.249 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:56:01.359 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345943 2024-09-15T05:56:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345943 2024-09-15T05:56:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509472 3-85899345943' 2024-09-15T05:56:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:01.360 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:56:01.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182420 2024-09-15T05:56:01.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182420 2024-09-15T05:56:01.470 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509472 3-85899345943 4-107374182420' 2024-09-15T05:56:01.471 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:01.471 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:56:01.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018890 2024-09-15T05:56:01.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018890 2024-09-15T05:56:01.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509472 3-85899345943 4-107374182420 5-128849018890' 2024-09-15T05:56:01.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:01.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836524 2024-09-15T05:56:01.583 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:01.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:56:01.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836524 2024-09-15T05:56:01.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:01.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836524 2024-09-15T05:56:01.585 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836524 2024-09-15T05:56:01.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-15T05:56:01.586 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:56:01.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-15T05:56:01.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:01.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949672996 2024-09-15T05:56:01.975 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:01.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:56:01.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949672996 2024-09-15T05:56:01.976 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:01.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949672996 2024-09-15T05:56:01.977 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949672996 2024-09-15T05:56:01.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-15T05:56:01.978 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:56:02.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-15T05:56:02.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:02.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509472 2024-09-15T05:56:02.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:02.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:56:02.373 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509472 2024-09-15T05:56:02.374 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:02.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509472 2024-09-15T05:56:02.375 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509472 2024-09-15T05:56:02.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-15T05:56:02.375 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:56:02.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-15T05:56:02.728 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:02.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345943 2024-09-15T05:56:02.729 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:02.730 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:56:02.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345943 2024-09-15T05:56:02.730 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:02.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345943 2024-09-15T05:56:02.731 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345943 2024-09-15T05:56:02.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2024-09-15T05:56:02.732 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:56:03.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345943 -lt 85899345943 2024-09-15T05:56:03.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:03.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182420 2024-09-15T05:56:03.089 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:03.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:56:03.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182420 2024-09-15T05:56:03.090 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182420 2024-09-15T05:56:03.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182420' 2024-09-15T05:56:03.092 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182420 2024-09-15T05:56:03.092 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:56:03.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182419 -lt 107374182420 2024-09-15T05:56:03.445 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:56:04.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:56:04.446 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:56:04.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182420 -lt 107374182420 2024-09-15T05:56:04.811 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:04.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018890 2024-09-15T05:56:04.812 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:04.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:56:04.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018890 2024-09-15T05:56:04.813 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:04.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018890 2024-09-15T05:56:04.814 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018890 2024-09-15T05:56:04.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-09-15T05:56:04.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:56:05.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018890 -lt 128849018890 2024-09-15T05:56:05.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:56:05.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:05.200 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:05.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:05.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:06.046 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:56:06.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:06.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:06.047 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:56:06.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-09-15T05:56:06.526 INFO:tasks.workunit.client.0.smithi138.stdout:writing initial objects 2024-09-15T05:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-09-15T05:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:06.526 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-09-15T05:56:06.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:06.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-09-15T05:56:06.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:06.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-09-15T05:56:06.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-09-15T05:56:07.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-09-15T05:56:07.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-09-15T05:56:07.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.401 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-09-15T05:56:07.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-09-15T05:56:07.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-09-15T05:56:07.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-09-15T05:56:07.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:07.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-09-15T05:56:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-09-15T05:56:08.182 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-09-15T05:56:08.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-09-15T05:56:08.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-09-15T05:56:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-09-15T05:56:08.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-09-15T05:56:08.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-09-15T05:56:08.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:08.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-09-15T05:56:09.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.132 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-09-15T05:56:09.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-09-15T05:56:09.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-09-15T05:56:09.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-09-15T05:56:09.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-09-15T05:56:09.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-09-15T05:56:09.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:09.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-09-15T05:56:10.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-09-15T05:56:10.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-09-15T05:56:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-09-15T05:56:10.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-09-15T05:56:10.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-09-15T05:56:10.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.711 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-09-15T05:56:10.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-09-15T05:56:10.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:10.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-09-15T05:56:11.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-09-15T05:56:11.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-09-15T05:56:11.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-09-15T05:56:11.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.476 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-09-15T05:56:11.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.618 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-09-15T05:56:11.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-09-15T05:56:11.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-09-15T05:56:11.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:11.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-09-15T05:56:12.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-09-15T05:56:12.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-09-15T05:56:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-09-15T05:56:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-09-15T05:56:12.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-09-15T05:56:12.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:12.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-09-15T05:56:13.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.112 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-09-15T05:56:13.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-09-15T05:56:13.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-09-15T05:56:13.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-09-15T05:56:13.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-09-15T05:56:13.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-09-15T05:56:13.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:13.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-09-15T05:56:14.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-09-15T05:56:14.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-09-15T05:56:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-09-15T05:56:14.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-09-15T05:56:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-09-15T05:56:14.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.683 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-09-15T05:56:14.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-09-15T05:56:14.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:14.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-09-15T05:56:15.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-09-15T05:56:15.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-09-15T05:56:15.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-09-15T05:56:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-09-15T05:56:15.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-09-15T05:56:15.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-09-15T05:56:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:15.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-09-15T05:56:16.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.027 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-09-15T05:56:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-09-15T05:56:16.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-09-15T05:56:16.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-09-15T05:56:16.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-09-15T05:56:16.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-09-15T05:56:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-09-15T05:56:16.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:16.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-09-15T05:56:17.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-09-15T05:56:17.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-09-15T05:56:17.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-09-15T05:56:17.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-09-15T05:56:17.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.833 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-09-15T05:56:17.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:17.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-09-15T05:56:18.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-09-15T05:56:18.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.311 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-09-15T05:56:18.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-09-15T05:56:18.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-09-15T05:56:18.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-09-15T05:56:18.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-09-15T05:56:18.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:18.983 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-09-15T05:56:19.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-09-15T05:56:19.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-09-15T05:56:19.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.403 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-09-15T05:56:19.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-09-15T05:56:19.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.663 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-09-15T05:56:19.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-09-15T05:56:19.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:19.907 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-09-15T05:56:20.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:20.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-09-15T05:56:20.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-09-15T05:56:20.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-09-15T05:56:20.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T05:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-15T05:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:56:20.295 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T05:56:20.296 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:56:20.296 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:56:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:56:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:56:20.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:56:20.349 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:56:20.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:56:20.731 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:56:20.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:20.733 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:56:20.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836530 2024-09-15T05:56:20.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836530 2024-09-15T05:56:20.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530' 2024-09-15T05:56:20.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:20.844 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:56:20.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673002 2024-09-15T05:56:20.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673002 2024-09-15T05:56:20.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673002' 2024-09-15T05:56:20.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:20.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:56:21.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509478 2024-09-15T05:56:21.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509478 2024-09-15T05:56:21.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509478' 2024-09-15T05:56:21.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:21.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:56:21.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345949 2024-09-15T05:56:21.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345949 2024-09-15T05:56:21.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509478 3-85899345949' 2024-09-15T05:56:21.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:21.176 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:56:21.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182426 2024-09-15T05:56:21.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182426 2024-09-15T05:56:21.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509478 3-85899345949 4-107374182426' 2024-09-15T05:56:21.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:21.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:56:21.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018896 2024-09-15T05:56:21.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018896 2024-09-15T05:56:21.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509478 3-85899345949 4-107374182426 5-128849018896' 2024-09-15T05:56:21.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:21.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836530 2024-09-15T05:56:21.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:21.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:56:21.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836530 2024-09-15T05:56:21.398 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:21.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836530 2024-09-15T05:56:21.399 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836530 2024-09-15T05:56:21.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-09-15T05:56:21.400 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:56:21.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836530 -lt 21474836530 2024-09-15T05:56:21.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:21.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673002 2024-09-15T05:56:21.765 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:21.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:56:21.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673002 2024-09-15T05:56:21.767 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:21.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673002 2024-09-15T05:56:21.768 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673002 2024-09-15T05:56:21.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-09-15T05:56:21.769 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:56:22.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673002 -lt 42949673002 2024-09-15T05:56:22.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:22.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509478 2024-09-15T05:56:22.142 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:22.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:56:22.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509478 2024-09-15T05:56:22.144 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:22.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509478 2024-09-15T05:56:22.145 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509478 2024-09-15T05:56:22.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509478' 2024-09-15T05:56:22.145 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:56:22.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509478 -lt 64424509478 2024-09-15T05:56:22.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:22.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345949 2024-09-15T05:56:22.501 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:22.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:56:22.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345949 2024-09-15T05:56:22.502 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:22.503 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345949 2024-09-15T05:56:22.503 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345949 2024-09-15T05:56:22.504 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345949' 2024-09-15T05:56:22.504 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:56:22.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345949 -lt 85899345949 2024-09-15T05:56:22.870 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:22.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182426 2024-09-15T05:56:22.870 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:22.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:56:22.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182426 2024-09-15T05:56:22.872 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:22.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182426 2024-09-15T05:56:22.873 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182426 2024-09-15T05:56:22.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-09-15T05:56:22.874 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:56:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182426 -lt 107374182426 2024-09-15T05:56:23.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:23.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018896 2024-09-15T05:56:23.305 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:23.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:56:23.306 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018896 2024-09-15T05:56:23.307 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:23.307 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018896 2024-09-15T05:56:23.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018896' 2024-09-15T05:56:23.308 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018896 2024-09-15T05:56:23.308 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:56:23.739 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018896 -lt 128849018896 2024-09-15T05:56:23.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:56:23.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:23.740 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:24.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:24.224 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:24.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:56:24.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:24.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:24.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:25.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:56:25.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:56:25.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:56:25.044 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-09-15T05:56:25.920 INFO:tasks.workunit.client.0.smithi138.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-09-15T05:56:26.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-15T05:56:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-15T05:56:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-15T05:56:27.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-15T05:56:28.062 INFO:tasks.workunit.client.0.smithi138.stdout: 2, 2024-09-15T05:56:28.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-09-15T05:56:28.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:56:28.063 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:56:28.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:56:28.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:56:28.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:56:28.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:56:28.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-09-15T05:56:28.707 INFO:tasks.workunit.client.0.smithi138.stderr:osd.2 is already down. 2024-09-15T05:56:28.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-15T05:56:30.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-15T05:56:30.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-15T05:56:30.721 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:56:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:30.854 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T05:56:30.855 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-09-15T05:56:30.856 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:56:30.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=4a953349-6f51-42e9-8113-5a23bc8bed9d --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.494363/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T05:56:30.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T05:56:30.881 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:30.880+0000 7f2d48121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:30.883 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:30.883+0000 7f2d48121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:30.884 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:30.884+0000 7f2d48121440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:56:31.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:56:31.223 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:56:31.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:56:31.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:56:31.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:56:31.398 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:31.397+0000 7f2d48121440 -1 Falling back to public interface 2024-09-15T05:56:31.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:56:32.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:56:32.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:56:32.757 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:56:32.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:56:32.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:56:32.758 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:56:33.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:56:34.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:56:34.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:56:34.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:56:34.142 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:56:34.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:56:34.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:56:34.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:56:35.258 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:35.256+0000 7f2d48121440 -1 osd.2 46 log_to_monitors true 2024-09-15T05:56:35.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:56:35.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:56:35.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:56:35.513 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:56:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:56:35.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:56:36.242 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/508706,v1:127.0.0.1:6819/508706] [v2:127.0.0.1:6820/508706,v1:127.0.0.1:6821/508706] exists,up 223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:56:36.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:56:36.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:56:36.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:56:36.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-09-15T05:56:36.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:56:36.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:56:36.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:56:36.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:56:36.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:56:36.246 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:56:36.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:56:36.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:56:36.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:56:36.760 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/508706,v1:127.0.0.1:6819/508706] [v2:127.0.0.1:6820/508706,v1:127.0.0.1:6821/508706] exists,up 223de97b-974f-44df-a396-2a12c0294c86 2024-09-15T05:56:36.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:56:36.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:56:36.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:56:36.760 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-09-15T05:56:38.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-09-15T05:56:38.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-09-15T05:56:38.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-09-15T05:56:38.893 INFO:tasks.workunit.client.0.smithi138.stdout: 2 2024-09-15T05:56:38.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T05:56:38.895 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T05:56:38.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-09-15T05:56:38.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T05:56:38.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T05:56:38.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T05:56:38.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T05:56:38.948 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T05:56:39.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T05:56:39.302 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T05:56:39.302 INFO:tasks.workunit.client.0.smithi138.stderr:2 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr:3 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr:4 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr:5' 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.303 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T05:56:39.460 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836535 2024-09-15T05:56:39.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836535 2024-09-15T05:56:39.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535' 2024-09-15T05:56:39.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.461 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T05:56:39.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=42949673008 2024-09-15T05:56:39.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 42949673008 2024-09-15T05:56:39.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-42949673008' 2024-09-15T05:56:39.573 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T05:56:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=214748364803 2024-09-15T05:56:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 214748364803 2024-09-15T05:56:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-42949673008 2-214748364803' 2024-09-15T05:56:39.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.737 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-15T05:56:39.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=85899345955 2024-09-15T05:56:39.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 85899345955 2024-09-15T05:56:39.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-42949673008 2-214748364803 3-85899345955' 2024-09-15T05:56:39.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-09-15T05:56:39.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=107374182431 2024-09-15T05:56:39.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 107374182431 2024-09-15T05:56:39.981 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-42949673008 2-214748364803 3-85899345955 4-107374182431' 2024-09-15T05:56:39.982 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T05:56:39.982 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-09-15T05:56:40.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=128849018902 2024-09-15T05:56:40.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 128849018902 2024-09-15T05:56:40.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836535 1-42949673008 2-214748364803 3-85899345955 4-107374182431 5-128849018902' 2024-09-15T05:56:40.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:40.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836535 2024-09-15T05:56:40.093 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:40.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T05:56:40.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836535 2024-09-15T05:56:40.094 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:40.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836535 2024-09-15T05:56:40.095 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836535 2024-09-15T05:56:40.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836535' 2024-09-15T05:56:40.095 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T05:56:40.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836535 -lt 21474836535 2024-09-15T05:56:40.527 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:40.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-42949673008 2024-09-15T05:56:40.527 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:40.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T05:56:40.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-42949673008 2024-09-15T05:56:40.528 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:40.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=42949673008 2024-09-15T05:56:40.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 42949673008' 2024-09-15T05:56:40.530 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 42949673008 2024-09-15T05:56:40.530 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:56:41.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673006 -lt 42949673008 2024-09-15T05:56:41.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T05:56:42.006 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T05:56:42.006 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T05:56:42.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 42949673008 -lt 42949673008 2024-09-15T05:56:42.367 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:42.367 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-214748364803 2024-09-15T05:56:42.368 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:42.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T05:56:42.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-214748364803 2024-09-15T05:56:42.369 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:42.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=214748364803 2024-09-15T05:56:42.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 214748364803' 2024-09-15T05:56:42.370 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 214748364803 2024-09-15T05:56:42.370 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T05:56:42.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 214748364804 -lt 214748364803 2024-09-15T05:56:42.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:42.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 3-85899345955 2024-09-15T05:56:42.842 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:42.842 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=3 2024-09-15T05:56:42.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 3-85899345955 2024-09-15T05:56:42.843 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:42.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=85899345955 2024-09-15T05:56:42.844 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.3 seq 85899345955 2024-09-15T05:56:42.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.3 seq 85899345955' 2024-09-15T05:56:42.845 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-15T05:56:43.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 85899345955 -lt 85899345955 2024-09-15T05:56:43.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:43.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 4-107374182431 2024-09-15T05:56:43.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:43.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=4 2024-09-15T05:56:43.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 4-107374182431 2024-09-15T05:56:43.432 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:43.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=107374182431 2024-09-15T05:56:43.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-09-15T05:56:43.433 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.4 seq 107374182431 2024-09-15T05:56:43.433 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 4 2024-09-15T05:56:44.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 107374182432 -lt 107374182431 2024-09-15T05:56:44.000 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T05:56:44.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 5-128849018902 2024-09-15T05:56:44.001 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T05:56:44.001 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=5 2024-09-15T05:56:44.002 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 5-128849018902 2024-09-15T05:56:44.003 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T05:56:44.003 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=128849018902 2024-09-15T05:56:44.004 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.5 seq 128849018902 2024-09-15T05:56:44.004 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.5 seq 128849018902' 2024-09-15T05:56:44.004 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 5 2024-09-15T05:56:44.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 128849018902 -lt 128849018902 2024-09-15T05:56:44.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T05:56:44.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:44.565 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:45.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:45.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:56:45.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:45.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:45.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:56:46.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:56:46.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:46.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:46.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:56:46.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:46.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:46.724 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:56:47.203 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:56:47.204 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:56:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1996 2024-09-15T05:56:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1996 '!=' null 2024-09-15T05:56:47.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:56:47.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:56:47.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:56:47.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:47.815 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:48.165 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T05:56:48.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:48.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:48.166 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:48.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T05:56:48.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T05:56:48.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T05:56:48.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T05:56:49.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1996 2024-09-15T05:56:49.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1996 '!=' null 2024-09-15T05:56:49.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-09-15T05:56:49.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T05:56:49.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T05:56:49.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T05:56:49.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T05:56:49.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T05:56:49.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T05:56:49.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T05:56:49.608 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T05:56:50.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T05:56:50.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T05:56:50.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T05:56:50.083 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-09-15T05:56:50.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.0.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log 2024-09-15T05:56:50.088 INFO:tasks.workunit.client.0.smithi138.stdout:success 2024-09-15T05:56:50.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-09-15T05:56:50.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-09-15T05:56:50.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-09-15T05:56:50.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-09-15T05:56:50.573 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' does not exist 2024-09-15T05:56:50.583 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-09-15T05:56:50.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:56:50.584 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:56:50.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:56:50.584 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:56:50.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-15T05:56:55.900 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:56:55.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:56:55.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:56:55.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:56:55.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:56:55.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:56:55.908 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:56:55.909 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:56:55.909 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:56:55.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:56:55.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:56:55.911 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:56:55.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:55.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:56:55.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:56:55.912 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:55.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:56:55.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:56:55.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-15T05:56:55.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:56:55.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:55.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:56:55.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494363 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/repeer-on-acting-back 0 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:56:55.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:56:55.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:56:55.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:56:55.956 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:56:55.956 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:56:55.957 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:56:55.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:56:55.958 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:56:55.958 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:56:55.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:55.959 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:56:55.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:56:55.960 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:55.960 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:56:55.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T05:56:55.962 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-09-15T05:56:55.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:56:55.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:55.963 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494363 2024-09-15T05:56:55.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494363 2024-09-15T05:56:55.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:56:55.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:56:55.965 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T05:56:55.965 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T05:56:55.966 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T05:56:55.992 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-09-15T05:56:55.992 DEBUG:teuthology.orchestra.run.smithi138:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=d75f23820806da622f561ee5ee7551015285993f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-09-15T05:56:56.055 INFO:tasks.workunit.client.0.smithi138.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-15T05:56:56.060 INFO:tasks.workunit.client.0.smithi138.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: export PYTHONWARNINGS=ignore 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2279: main: PYTHONWARNINGS=ignore 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: export CEPH_CONF=/dev/null 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: main: CEPH_CONF=/dev/null 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: main: unset CEPH_ARGS 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: main: local code 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: main: run td/repro-long-log 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-09-15T05:56:56.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-09-15T05:56:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-09-15T05:56:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-09-15T05:56:56.062 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-09-15T05:56:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none ' 2024-09-15T05:56:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_repro_long_log2 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_trim_max_entries 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:TEST_trim_max_entries_with_dups' 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-15T05:56:56.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:56:56.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:56:56.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:56:56.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:56:56.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:56:56.066 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:56:56.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:56:56.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:56:56.067 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:56:56.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:56.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:56:56.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:56:56.068 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:56:56.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:56:56.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:56:56.070 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T05:56:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:56:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:56.071 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:56.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T05:56:56.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:56:56.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:56:56.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-15T05:56:56.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:56:56.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:56.074 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:56.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.510179 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-15T05:56:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-15T05:56:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-15T05:56:56.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:56:56.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:56:56.447 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:56:56.447 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:56:56.447 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:56.448 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:56.448 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:56:56.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:56:56.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:56.692 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:56.693 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T05:56:56.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:56:56.693 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get fsid 2024-09-15T05:56:56.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:56:56.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:56:56.773 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:56:56.774 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get mon_host 2024-09-15T05:56:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-15T05:56:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-15T05:56:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:56:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:56:56.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:56:56.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-15T05:56:56.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:56:57.675 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:56:57.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:56:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-15T05:56:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T05:56:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:56:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:56:57.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:56:57.701 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:56:57.702 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-15T05:56:57.703 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:56:57.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c97d04e8-424c-491b-b246-69e60b60259a 2024-09-15T05:56:57.704 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c97d04e8-424c-491b-b246-69e60b60259a' 2024-09-15T05:56:57.705 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 c97d04e8-424c-491b-b246-69e60b60259a 2024-09-15T05:56:57.705 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:56:57.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApd+ZmoqGGKhAAhI2KetSZCdKdsfTas68xuw== 2024-09-15T05:56:57.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApd+ZmoqGGKhAAhI2KetSZCdKdsfTas68xuw=="}' 2024-09-15T05:56:57.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c97d04e8-424c-491b-b246-69e60b60259a -i td/repro-long-log/0/new.json 2024-09-15T05:56:57.978 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:56:57.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-15T05:56:57.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApd+ZmoqGGKhAAhI2KetSZCdKdsfTas68xuw== --osd-uuid c97d04e8-424c-491b-b246-69e60b60259a 2024-09-15T05:56:58.011 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.010+0000 7ff228271440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:58.025 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.024+0000 7ff228271440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:58.026 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.025+0000 7ff228271440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:56:58.026 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.025+0000 7ff228271440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:56:58.026 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.025+0000 7ff228271440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:56:58.026 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.025+0000 7ff228271440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:56:58.065 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:56:58.064+0000 7ff228271440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-15T05:57:03.424 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-15T05:57:03.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:57:03.425 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:57:03.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:57:03.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:57:03.907 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:57:03.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:57:03.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:57:03.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:57:03.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:57:03.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:57:03.928 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:03.927+0000 7f74e5188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:03.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:03.929+0000 7f74e5188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:03.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:03.930+0000 7f74e5188440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:04.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:57:04.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:57:04.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:57:04.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:57:04.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:57:04.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:04.253 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:57:04.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:57:04.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:04.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:04.455 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:04.455+0000 7f74e5188440 -1 Falling back to public interface 2024-09-15T05:57:04.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:05.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:05.627 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:05.628 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:57:05.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:57:05.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:05.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:05.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:06.923 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:06.922+0000 7f74e5188440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:57:06.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:06.993 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:06.993 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:57:06.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:57:06.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:06.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:07.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:08.543 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:08.543 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:57:08.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:57:08.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:08.544 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:08.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:09.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:09.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:09.938 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:57:09.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:57:09.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:09.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:10.299 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:11.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:11.301 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:11.301 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:57:11.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:57:11.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:11.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:11.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:12.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:12.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:12.750 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:57:12.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:57:12.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:12.751 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:13.105 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:14.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:14.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:14.107 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:57:14.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:57:14.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:14.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:14.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:15.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:15.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:15.468 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:57:15.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:57:15.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:15.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:15.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:16.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:16.931 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:16.931 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:57:16.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:57:16.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:16.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:17.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:18.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:18.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:18.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:57:18.304 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:57:18.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:18.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:19.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:20.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:20.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:20.334 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:57:20.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:57:20.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:20.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:20.883 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:21.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:21.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:21.885 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:57:21.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:57:21.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:21.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:22.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:23.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:23.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:23.642 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:57:23.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:57:23.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:23.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:24.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:25.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:25.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:25.734 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:57:25.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:57:25.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:25.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:26.207 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:27.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:27.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:27.209 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:57:27.209 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:57:27.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:27.210 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:27.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:28.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:28.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:28.913 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:57:28.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:57:28.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:28.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:57:29.322 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/511270,v1:127.0.0.1:6803/511270] [v2:127.0.0.1:6804/511270,v1:127.0.0.1:6805/511270] exists,up c97d04e8-424c-491b-b246-69e60b60259a 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T05:57:29.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T05:57:29.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T05:57:29.328 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:57:29.329 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 cd45685b-0e3c-4f5f-b8b3-c916050ca4f5 2024-09-15T05:57:29.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cd45685b-0e3c-4f5f-b8b3-c916050ca4f5 2024-09-15T05:57:29.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 cd45685b-0e3c-4f5f-b8b3-c916050ca4f5' 2024-09-15T05:57:29.330 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:57:29.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJd+ZmPpUnFBAAow42cq5qrV/QGzIN+i+3hA== 2024-09-15T05:57:29.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJd+ZmPpUnFBAAow42cq5qrV/QGzIN+i+3hA=="}' 2024-09-15T05:57:29.339 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cd45685b-0e3c-4f5f-b8b3-c916050ca4f5 -i td/repro-long-log/1/new.json 2024-09-15T05:57:29.782 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:57:29.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-15T05:57:29.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJd+ZmPpUnFBAAow42cq5qrV/QGzIN+i+3hA== --osd-uuid cd45685b-0e3c-4f5f-b8b3-c916050ca4f5 2024-09-15T05:57:29.816 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.815+0000 7f7f5980b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:29.817 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.817+0000 7f7f5980b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:29.818 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.818+0000 7f7f5980b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:29.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.818+0000 7f7f5980b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:57:29.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.818+0000 7f7f5980b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:57:29.819 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.818+0000 7f7f5980b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:57:29.897 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:29.897+0000 7f7f5980b440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-15T05:57:35.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-15T05:57:35.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:57:35.715 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:57:35.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:57:35.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:57:36.207 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:57:36.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:57:36.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:57:36.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:57:36.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:57:36.208 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:57:36.228 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:36.226+0000 7fd89542a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:36.229 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:36.228+0000 7fd89542a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:36.230 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:36.229+0000 7fd89542a440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:57:36.565 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:57:36.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:36.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:36.750 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:36.749+0000 7fd89542a440 -1 Falling back to public interface 2024-09-15T05:57:36.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:37.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:37.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:37.916 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:57:37.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:57:37.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:37.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:39.191 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:57:39.190+0000 7fd89542a440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:57:39.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:39.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:39.271 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:57:39.271 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:57:39.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:39.272 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:39.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:40.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:40.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:40.853 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:57:40.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:57:40.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:40.854 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:41.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:42.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:42.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:42.223 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:57:42.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:57:42.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:42.224 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:42.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:43.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:43.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:43.612 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:57:43.612 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:57:43.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:43.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:44.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:45.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:45.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:45.060 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:57:45.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:57:45.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:45.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:45.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:46.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:46.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:46.449 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:57:46.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:57:46.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:46.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:46.932 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:47.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:47.933 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:47.933 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:57:47.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:57:47.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:47.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:48.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:49.450 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:49.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:49.451 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:57:49.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:57:49.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:49.452 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:49.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:50.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:50.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:50.989 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:57:50.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:57:50.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:50.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:51.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:52.650 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:57:52.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:57:52.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:52.651 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:53.352 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:54.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:54.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:54.355 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:57:54.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:57:54.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:54.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:55.177 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:56.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:56.179 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:57:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:57:56.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:56.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:56.945 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:57.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:57.946 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:57.947 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:57:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:57:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:57.947 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:57:58.731 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:57:59.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:57:59.732 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:57:59.732 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:57:59.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:57:59.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:57:59.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:58:00.358 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:01.360 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:58:01.360 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:58:01.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:01.361 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:58:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:02.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:02.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:02.725 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:58:02.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:58:02.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:02.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:58:03.122 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/513000,v1:127.0.0.1:6811/513000] [v2:127.0.0.1:6812/513000,v1:127.0.0.1:6813/513000] exists,up cd45685b-0e3c-4f5f-b8b3-c916050ca4f5 2024-09-15T05:58:03.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:58:03.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:58:03.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:58:03.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-15T05:58:03.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:58:03.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:03.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:58:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5889fb65-5320-48bf-bd08-6ffab8bf29c8 2024-09-15T05:58:03.130 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 5889fb65-5320-48bf-bd08-6ffab8bf29c8 2024-09-15T05:58:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 5889fb65-5320-48bf-bd08-6ffab8bf29c8' 2024-09-15T05:58:03.131 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:58:03.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrd+ZmQDhBCBAABOaX/uvE8Z0kfuxUWvRl0w== 2024-09-15T05:58:03.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrd+ZmQDhBCBAABOaX/uvE8Z0kfuxUWvRl0w=="}' 2024-09-15T05:58:03.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5889fb65-5320-48bf-bd08-6ffab8bf29c8 -i td/repro-long-log/2/new.json 2024-09-15T05:58:03.622 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:58:03.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-15T05:58:03.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrd+ZmQDhBCBAABOaX/uvE8Z0kfuxUWvRl0w== --osd-uuid 5889fb65-5320-48bf-bd08-6ffab8bf29c8 2024-09-15T05:58:03.656 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.655+0000 7fe87b325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:03.657 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.657+0000 7fe87b325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:03.658 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.658+0000 7fe87b325440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:03.659 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.658+0000 7fe87b325440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T05:58:03.659 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.658+0000 7fe87b325440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T05:58:03.659 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.658+0000 7fe87b325440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T05:58:03.738 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:03.737+0000 7fe87b325440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-15T05:58:09.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-15T05:58:09.595 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:58:09.596 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T05:58:09.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T05:58:09.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:58:10.082 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T05:58:10.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T05:58:10.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:58:10.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:58:10.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:58:10.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:58:10.102 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:10.101+0000 7fcc9e810440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:10.104 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:10.103+0000 7fcc9e810440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:10.105 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:10.104+0000 7fcc9e810440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:10.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:58:10.432 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:10.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:10.433 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:10.618 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:10.618+0000 7fcc9e810440 -1 Falling back to public interface 2024-09-15T05:58:10.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:11.795 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:58:11.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:58:11.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:11.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:12.196 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:13.134 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:13.133+0000 7fcc9e810440 -1 osd.2 0 log_to_monitors true 2024-09-15T05:58:13.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:13.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:13.198 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:58:13.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:58:13.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:13.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:13.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:14.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:14.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:14.632 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:58:14.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:58:14.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:14.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:15.065 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:16.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:16.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:16.067 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:58:16.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:58:16.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:16.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:16.446 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:17.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:17.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:17.448 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:58:17.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:58:17.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:17.449 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:17.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:18.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:18.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:18.940 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:58:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:58:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:18.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:19.354 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:20.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:20.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:20.356 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:58:20.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:58:20.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:20.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:20.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:21.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:21.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:21.749 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:58:21.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:58:21.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:21.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:22.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:23.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:23.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:23.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:58:23.266 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:58:23.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:23.267 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:23.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:24.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:24.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:24.678 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:58:24.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:58:24.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:24.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:25.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:26.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:26.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:26.081 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:58:26.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:58:26.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:26.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:26.834 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:27.835 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:27.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:27.836 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:58:27.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:58:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:28.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:29.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:29.689 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:29.690 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:58:29.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:58:29.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:29.690 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:30.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:31.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:31.442 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:58:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:58:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:31.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:31.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:32.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:32.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:32.895 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:58:32.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:58:32.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:32.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:33.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:34.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:34.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:34.634 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:58:34.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:58:34.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:34.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:35.451 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:36.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:36.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:36.453 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:58:36.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:58:36.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:36.454 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:36.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:58:37.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:58:37.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:58:37.873 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T05:58:37.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-15T05:58:37.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:58:37.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T05:58:38.268 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/514754,v1:127.0.0.1:6819/514754] [v2:127.0.0.1:6820/514754,v1:127.0.0.1:6821/514754] exists,up 5889fb65-5320-48bf-bd08-6ffab8bf29c8 2024-09-15T05:58:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:58:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:58:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:58:38.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-15T05:58:38.826 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T05:58:38.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-15T05:58:38.838 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-15T05:58:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-15T05:58:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-15T05:58:39.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-15T05:58:39.350 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T05:58:39.376 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T05:58:39.401 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T05:58:39.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-15T05:58:39.534 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T05:58:39.560 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T05:58:39.585 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T05:58:39.593 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-15T05:58:39.690 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T05:58:39.715 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T05:58:39.739 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T05:58:39.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-15T05:58:39.845 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T05:58:39.871 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T05:58:39.897 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T05:58:39.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-15T05:58:40.002 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T05:58:40.027 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T05:58:40.051 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T05:58:40.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-15T05:58:40.061 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-15T05:58:40.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:40.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:40.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:40.786 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.168 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.169 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.385 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.485 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.828 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:41.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.172 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.766 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:42.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:42.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-15T05:58:42.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T05:58:42.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-15T05:58:42.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T05:58:42.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T05:58:43.071 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836504 2024-09-15T05:58:43.092 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672974 2024-09-15T05:58:43.113 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509447 2024-09-15T05:58:43.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T05:58:46.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:46.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:46.242 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T05:58:46.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:46.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:46.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-15T05:58:46.349 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T05:58:46.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:46.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:46.458 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:46.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:46.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:46.459 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T05:58:46.565 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:46.566 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-15T05:58:46.661 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-15T05:58:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:46.662 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:46.801 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:46.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:46.803 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:46.893 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:46.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:46.985 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:46.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:46.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.102 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.104 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.211 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.304 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.396 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.488 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.597 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.598 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.706 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.824 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.826 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:47.982 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:47.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:47.984 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.066 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.067 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.068 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.150 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.151 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.258 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.342 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.344 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.460 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.461 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.552 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.644 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T05:58:48.645 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T05:58:48.761 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T05:58:48.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-15T05:58:48.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T05:58:48.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-15T05:58:48.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T05:58:48.763 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T05:58:48.856 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836507 2024-09-15T05:58:48.878 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672977 2024-09-15T05:58:48.899 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509450 2024-09-15T05:58:48.908 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T05:58:51.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:51.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:52.028 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T05:58:52.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:52.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:52.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-15T05:58:52.136 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T05:58:52.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:52.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:52.244 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:52.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:52.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:52.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T05:58:52.352 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:52.353 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-09-15T05:58:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-09-15T05:58:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T05:58:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-09-15T05:58:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T05:58:52.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T05:58:52.585 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836510 2024-09-15T05:58:52.606 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672979 2024-09-15T05:58:52.628 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509453 2024-09-15T05:58:52.636 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T05:58:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:55.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:55.756 INFO:tasks.workunit.client.0.smithi138.stdout:22 2024-09-15T05:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T05:58:55.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-09-15T05:58:55.905 INFO:tasks.workunit.client.0.smithi138.stdout:22 2024-09-15T05:58:55.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:55.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:56.013 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:56.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T05:58:56.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T05:58:56.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T05:58:56.121 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:58:56.122 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:58:56.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:58:56.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:58:56.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:58:56.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:58:56.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:58:56.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:58:56.230 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:58:56.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:58:56.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:58:56.232 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:58:56.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:58:56.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:58:56.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:58:56.233 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:58:56.233 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:58:56.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:58:56.235 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T05:58:56.249 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:58:56.249 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.249 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T05:58:56.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:58:56.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:58:56.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-15T05:58:56.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T05:58:56.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T05:58:56.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T05:58:56.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T05:58:56.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T05:58:56.253 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T05:58:56.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T05:58:56.254 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T05:58:56.254 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T05:58:56.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:58:56.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T05:58:56.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T05:58:56.256 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T05:58:56.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T05:58:56.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T05:58:56.257 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T05:58:56.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T05:58:56.258 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.259 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T05:58:56.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T05:58:56.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T05:58:56.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-15T05:58:56.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T05:58:56.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.261 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.510179 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-09-15T05:58:56.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-15T05:58:56.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-15T05:58:56.623 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:56.624 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T05:58:56.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:58:56.876 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.877 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.877 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T05:58:56.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:58:56.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get fsid 2024-09-15T05:58:56.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T05:58:56.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T05:58:56.953 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get mon_host 2024-09-15T05:58:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-15T05:58:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-15T05:58:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T05:58:57.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T05:58:57.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T05:58:57.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-15T05:58:57.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T05:58:57.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T05:58:57.877 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:58:57.878 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T05:58:57.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-15T05:58:57.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:58:57.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-15T05:58:57.905 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:58:57.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c9ee76d2-da51-4d3a-8206-7119b4b92c49 2024-09-15T05:58:57.906 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 c9ee76d2-da51-4d3a-8206-7119b4b92c49 2024-09-15T05:58:57.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c9ee76d2-da51-4d3a-8206-7119b4b92c49' 2024-09-15T05:58:57.906 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:58:57.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQChd+ZmhP6TNhAAoTK6B+e9WNYc6JNz0u1kPg== 2024-09-15T05:58:57.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQChd+ZmhP6TNhAAoTK6B+e9WNYc6JNz0u1kPg=="}' 2024-09-15T05:58:57.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c9ee76d2-da51-4d3a-8206-7119b4b92c49 -i td/repro-long-log/0/new.json 2024-09-15T05:58:58.222 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:58:58.231 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-15T05:58:58.232 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChd+ZmhP6TNhAAoTK6B+e9WNYc6JNz0u1kPg== --osd-uuid c9ee76d2-da51-4d3a-8206-7119b4b92c49 2024-09-15T05:58:58.255 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.254+0000 7f99dd015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:58.268 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.267+0000 7f99dd015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.268+0000 7f99dd015440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:58:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.268+0000 7f99dd015440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:58:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.268+0000 7f99dd015440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:58:58.269 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.268+0000 7f99dd015440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T05:58:58.308 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:58:58.307+0000 7f99dd015440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-15T05:59:03.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-15T05:59:03.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:59:03.586 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T05:59:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T05:59:03.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:59:04.065 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T05:59:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T05:59:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:59:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:59:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:59:04.066 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:59:04.087 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:04.086+0000 7f594d357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:04.089 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:04.088+0000 7f594d357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:04.090 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:04.089+0000 7f594d357440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:04.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T05:59:04.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:59:04.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T05:59:04.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:04.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:04.612 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:04.612+0000 7f594d357440 -1 Falling back to public interface 2024-09-15T05:59:04.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:05.772 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:05.772 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:59:05.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:05.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:59:05.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:05.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:06.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:07.053 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:07.051+0000 7f594d357440 -1 osd.0 0 log_to_monitors true 2024-09-15T05:59:07.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:07.125 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:59:07.125 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:07.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:59:07.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:07.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:07.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:08.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:08.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:08.781 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:59:08.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:59:08.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:08.781 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:09.173 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:10.174 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:10.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:10.175 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:59:10.175 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:59:10.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:10.176 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:10.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:11.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:11.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:11.547 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:59:11.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:59:11.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:11.548 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:11.968 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:12.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:12.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:12.970 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:59:12.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:59:12.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:12.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:13.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:14.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:14.349 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:59:14.349 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:14.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:59:14.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:14.350 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:14.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:15.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:15.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:15.886 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:59:15.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:59:15.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:15.886 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:16.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:17.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:17.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:17.294 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:59:17.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:59:17.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:17.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:17.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:18.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:18.653 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:18.653 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:59:18.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:59:18.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:18.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:19.644 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:20.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:20.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:20.646 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:59:20.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:59:20.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:20.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:21.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:22.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:22.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:22.119 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:59:22.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:59:22.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:22.119 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:22.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:23.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:23.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:23.809 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:59:23.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:59:23.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:23.809 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:24.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:25.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:25.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:25.922 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:59:25.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:59:25.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:25.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:26.362 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:27.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:27.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:27.363 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T05:59:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T05:59:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:27.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:28.089 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:29.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:29.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:29.091 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T05:59:29.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T05:59:29.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:29.091 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:29.498 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:30.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:30.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:30.500 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T05:59:30.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T05:59:30.500 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:30.501 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T05:59:30.846 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/518240,v1:127.0.0.1:6803/518240] [v2:127.0.0.1:6804/518240,v1:127.0.0.1:6805/518240] exists,up c9ee76d2-da51-4d3a-8206-7119b4b92c49 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-15T05:59:30.850 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T05:59:30.851 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T05:59:30.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T05:59:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-15T05:59:30.853 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T05:59:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=992a64b1-cf7e-4acc-baab-f0f9d938281c 2024-09-15T05:59:30.853 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 992a64b1-cf7e-4acc-baab-f0f9d938281c 2024-09-15T05:59:30.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 992a64b1-cf7e-4acc-baab-f0f9d938281c' 2024-09-15T05:59:30.854 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T05:59:30.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCd+ZmSNheMxAAODrG1mevBldjdqiaNviyXg== 2024-09-15T05:59:30.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCd+ZmSNheMxAAODrG1mevBldjdqiaNviyXg=="}' 2024-09-15T05:59:30.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 992a64b1-cf7e-4acc-baab-f0f9d938281c -i td/repro-long-log/1/new.json 2024-09-15T05:59:31.323 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:59:31.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-15T05:59:31.335 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCd+ZmSNheMxAAODrG1mevBldjdqiaNviyXg== --osd-uuid 992a64b1-cf7e-4acc-baab-f0f9d938281c 2024-09-15T05:59:31.357 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.356+0000 7fb388d27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:31.359 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.358+0000 7fb388d27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:31.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.359+0000 7fb388d27440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:31.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.359+0000 7fb388d27440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:59:31.360 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.359+0000 7fb388d27440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:59:31.361 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.359+0000 7fb388d27440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T05:59:31.413 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:31.413+0000 7fb388d27440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-15T05:59:37.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-15T05:59:37.011 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T05:59:37.011 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T05:59:37.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T05:59:37.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T05:59:37.505 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T05:59:37.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T05:59:37.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T05:59:37.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T05:59:37.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T05:59:37.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T05:59:37.527 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:37.526+0000 7f947ef4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:37.554 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:37.553+0000 7f947ef4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:37.555 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:37.554+0000 7f947ef4f440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T05:59:37.895 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:37.896 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T05:59:37.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T05:59:37.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:37.896 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:38.092 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:38.091+0000 7f947ef4f440 -1 Falling back to public interface 2024-09-15T05:59:38.393 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:39.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:39.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:39.394 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T05:59:39.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T05:59:39.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:39.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:39.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:40.658 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T05:59:40.657+0000 7f947ef4f440 -1 osd.1 0 log_to_monitors true 2024-09-15T05:59:40.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:40.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:40.795 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T05:59:40.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T05:59:40.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:40.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:41.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:42.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:42.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:42.215 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T05:59:42.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T05:59:42.215 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:42.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:42.604 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:43.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:43.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:43.606 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T05:59:43.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T05:59:43.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:43.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:43.985 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:44.986 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:44.987 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T05:59:44.987 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T05:59:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:44.988 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:45.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:46.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:46.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:46.428 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T05:59:46.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T05:59:46.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:46.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:46.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:47.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:47.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:47.845 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T05:59:47.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T05:59:47.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:47.845 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:48.303 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:49.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:49.304 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:49.304 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T05:59:49.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T05:59:49.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:49.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:49.717 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:50.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:50.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:50.719 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T05:59:50.719 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T05:59:50.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:50.720 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:51.106 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:52.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:52.107 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T05:59:52.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T05:59:52.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:52.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:52.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:53.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:53.532 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:53.532 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T05:59:53.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T05:59:53.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:53.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:54.488 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:55.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:55.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:55.489 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T05:59:55.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T05:59:55.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:55.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:55.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:56.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:56.924 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T05:59:56.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T05:59:56.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:56.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:57.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T05:59:58.641 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T05:59:58.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T05:59:58.642 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T05:59:58.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T05:59:58.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T05:59:58.643 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T05:59:59.406 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:00.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:00.407 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:00.407 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:00:00.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:00:00.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:00.408 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:01.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:02.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:02.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:02.587 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:00:02.588 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:00:02.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:02.589 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:03.313 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:04.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:04.314 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:04.314 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:00:04.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:00:04.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:04.315 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:04.660 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/519995,v1:127.0.0.1:6811/519995] [v2:127.0.0.1:6812/519995,v1:127.0.0.1:6813/519995] exists,up 992a64b1-cf7e-4acc-baab-f0f9d938281c 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:00:04.668 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:00:04.669 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:00:04.670 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:00:04.671 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:00:04.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:00:04.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d02bdd56-dcbe-4aa4-aeae-b26ba07cc909 2024-09-15T06:00:04.674 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 d02bdd56-dcbe-4aa4-aeae-b26ba07cc909 2024-09-15T06:00:04.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d02bdd56-dcbe-4aa4-aeae-b26ba07cc909' 2024-09-15T06:00:04.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:00:04.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkd+ZmK6ZLKBAATfPlZL1zxbqPhNpqqMBjdA== 2024-09-15T06:00:04.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkd+ZmK6ZLKBAATfPlZL1zxbqPhNpqqMBjdA=="}' 2024-09-15T06:00:04.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d02bdd56-dcbe-4aa4-aeae-b26ba07cc909 -i td/repro-long-log/2/new.json 2024-09-15T06:00:05.217 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:00:05.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-15T06:00:05.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkd+ZmK6ZLKBAATfPlZL1zxbqPhNpqqMBjdA== --osd-uuid d02bdd56-dcbe-4aa4-aeae-b26ba07cc909 2024-09-15T06:00:05.251 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.250+0000 7fef47d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:05.253 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.252+0000 7fef47d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:05.253 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.253+0000 7fef47d21440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:05.254 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.253+0000 7fef47d21440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:00:05.254 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.253+0000 7fef47d21440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:00:05.254 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.253+0000 7fef47d21440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:00:05.341 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:05.340+0000 7fef47d21440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-15T06:00:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-15T06:00:11.291 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:00:11.292 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T06:00:11.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T06:00:11.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:00:11.775 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T06:00:11.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T06:00:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:00:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:00:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:00:11.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:00:11.795 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:11.794+0000 7f6a4ed1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:11.797 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:11.796+0000 7f6a4ed1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:11.798 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:11.797+0000 7f6a4ed1e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:00:12.157 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:12.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:12.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:12.312 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:12.312+0000 7f6a4ed1e440 -1 Falling back to public interface 2024-09-15T06:00:12.511 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:13.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:13.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:13.513 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:00:13.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:00:13.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:13.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:13.872 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:14.785 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:14.784+0000 7f6a4ed1e440 -1 osd.2 0 log_to_monitors true 2024-09-15T06:00:14.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:14.873 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:14.874 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:00:14.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:00:14.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:14.874 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:15.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:16.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:16.442 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:16.442 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:00:16.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:00:16.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:16.443 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:16.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:17.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:17.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:17.901 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:00:17.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:00:17.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:17.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:18.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:19.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:19.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:19.411 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:00:19.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:00:19.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:19.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:19.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:20.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:00:20.903 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:00:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:20.903 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:21.281 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:22.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:22.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:22.283 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:00:22.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:00:22.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:22.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:22.789 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:23.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:23.791 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:23.791 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:00:23.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:00:23.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:23.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:24.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:25.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:00:25.218 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:00:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:25.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:25.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:26.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:26.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:26.756 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:00:26.756 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:00:26.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:26.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:27.107 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:28.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:28.108 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:28.108 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:00:28.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:00:28.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:28.109 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:28.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:29.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:29.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:29.806 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:00:29.806 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:00:29.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:29.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:30.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:31.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:31.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:31.661 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:00:31.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:00:31.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:31.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:32.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:33.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:33.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:33.414 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:00:33.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:00:33.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:33.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:34.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:35.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:35.294 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:00:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:00:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:35.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:36.069 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:37.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:37.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:37.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:00:37.071 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:00:37.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:37.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:37.750 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:38.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:38.752 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:38.752 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:00:38.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:00:38.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:38.753 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:00:39.110 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/521749,v1:127.0.0.1:6819/521749] [v2:127.0.0.1:6820/521749,v1:127.0.0.1:6821/521749] exists,up d02bdd56-dcbe-4aa4-aeae-b26ba07cc909 2024-09-15T06:00:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:00:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:00:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:00:39.110 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-15T06:00:39.706 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T06:00:39.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-15T06:00:39.718 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-15T06:00:40.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-15T06:00:40.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-15T06:00:40.143 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-15T06:00:40.283 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:00:40.309 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:00:40.334 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:00:40.343 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-15T06:00:40.440 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:00:40.466 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:00:40.490 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:00:40.499 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-15T06:00:40.678 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:00:40.703 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:00:40.728 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:00:40.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-15T06:00:40.935 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:00:40.960 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:00:40.985 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:00:40.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-15T06:00:41.140 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:00:41.165 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:00:41.190 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:00:41.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-15T06:00:41.199 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-15T06:00:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:41.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:41.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:41.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:41.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:41.967 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.137 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.336 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.722 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:42.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:42.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:43.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:43.170 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:43.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:43.355 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:43.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:00:43.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-15T06:00:43.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:00:43.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-15T06:00:43.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:00:43.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:00:43.658 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836502 2024-09-15T06:00:43.680 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672975 2024-09-15T06:00:43.701 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509445 2024-09-15T06:00:43.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:00:46.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:46.712 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:00:46.830 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:00:46.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:46.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:00:46.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-15T06:00:46.940 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:00:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:00:47.046 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:47.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:47.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:00:47.047 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:00:47.153 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:47.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.243 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-15T06:00:47.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.243 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.332 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.334 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.451 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.453 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.584 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.585 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.586 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.693 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.694 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.777 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.869 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.871 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:47.995 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:47.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:47.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.137 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.221 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.356 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.440 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.441 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.532 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.616 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.724 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.809 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:48.917 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:48.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:48.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:49.035 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:49.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:49.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:49.152 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:49.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:49.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:49.236 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:49.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:00:49.238 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:00:49.328 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:00:49.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-15T06:00:49.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:00:49.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-15T06:00:49.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:00:49.330 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:00:49.425 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836505 2024-09-15T06:00:49.447 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672979 2024-09-15T06:00:49.468 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509449 2024-09-15T06:00:49.477 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:00:52.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:52.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:00:52.598 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:00:52.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:52.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:00:52.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-15T06:00:52.707 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:00:52.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:52.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:00:52.815 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:52.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:00:52.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:00:52.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:00:52.923 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:52.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-09-15T06:00:52.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-09-15T06:00:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-09-15T06:00:53.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-09-15T06:00:53.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:00:53.032 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:00:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:00:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:00:53.032 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:00:53.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:00:53.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-09-15T06:00:53.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-09-15T06:00:54.750 INFO:tasks.workunit.client.0.smithi138.stderr:Log bounds are: (19'20,19'41] 2024-09-15T06:00:54.750 INFO:tasks.workunit.client.0.smithi138.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-09-15T06:00:54.783 INFO:tasks.workunit.client.0.smithi138.stderr:Finished trimming, now compacting... 2024-09-15T06:00:54.784 INFO:tasks.workunit.client.0.smithi138.stdout:Finished trimming pg log 2024-09-15T06:00:56.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-09-15T06:00:56.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-09-15T06:00:56.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-15T06:00:56.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-15T06:00:56.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-15T06:00:56.020 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:00:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:00:56.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:00:56.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-15T06:00:56.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-15T06:00:56.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-09-15T06:00:56.022 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-15T06:00:56.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-15T06:00:56.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-15T06:00:56.042 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:56.041+0000 7f7c6c447440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:56.076 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:56.075+0000 7f7c6c447440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:56.077 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:56.076+0000 7f7c6c447440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:00:56.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:56.375 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:00:56.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:00:56.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:56.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:56.598 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:56.598+0000 7f7c6c447440 -1 Falling back to public interface 2024-09-15T06:00:56.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:57.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:57.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:57.725 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:00:57.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:00:57.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:57.725 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:58.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:00:58.930 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:00:58.929+0000 7f7c6c447440 -1 osd.1 19 log_to_monitors true 2024-09-15T06:00:59.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:00:59.100 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:00:59.101 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:00:59.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:00:59.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:00:59.101 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:00:59.543 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 23 up_thru 17 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/524374,v1:127.0.0.1:6811/524374] [v2:127.0.0.1:6812/524374,v1:127.0.0.1:6813/524374] exists,up 992a64b1-cf7e-4acc-baab-f0f9d938281c 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-15T06:00:59.545 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-15T06:00:59.546 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-15T06:00:59.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-15T06:00:59.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-15T06:00:59.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-15T06:00:59.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-15T06:00:59.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: local timeout=300 2024-09-15T06:00:59.629 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ceph osd ls 2024-09-15T06:01:00.015 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: ids='0 2024-09-15T06:01:00.016 INFO:tasks.workunit.client.0.smithi138.stderr:1 2024-09-15T06:01:00.016 INFO:tasks.workunit.client.0.smithi138.stderr:2' 2024-09-15T06:01:00.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: seqs= 2024-09-15T06:01:00.016 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T06:01:00.016 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-15T06:01:00.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=21474836509 2024-09-15T06:01:00.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 21474836509 2024-09-15T06:01:00.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509' 2024-09-15T06:01:00.118 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T06:01:00.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-15T06:01:00.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=98784247811 2024-09-15T06:01:00.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 98784247811 2024-09-15T06:01:00.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-98784247811' 2024-09-15T06:01:00.221 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2172: flush_pg_stats: for osd in $ids 2024-09-15T06:01:00.221 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2173: flush_pg_stats: seq=64424509453 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2174: flush_pg_stats: test -z 64424509453 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2178: flush_pg_stats: seqs=' 0-21474836509 1-98784247811 2-64424509453' 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 0-21474836509 2024-09-15T06:01:00.324 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T06:01:00.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=0 2024-09-15T06:01:00.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 0-21474836509 2024-09-15T06:01:00.325 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T06:01:00.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=21474836509 2024-09-15T06:01:00.327 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.0 seq 21474836509 2024-09-15T06:01:00.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-09-15T06:01:00.327 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T06:01:00.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836507 -lt 21474836509 2024-09-15T06:01:00.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: flush_pg_stats: sleep 1 2024-09-15T06:01:01.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-15T06:01:01.674 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-15T06:01:02.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 21474836510 -lt 21474836509 2024-09-15T06:01:02.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T06:01:02.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 1-98784247811 2024-09-15T06:01:02.072 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T06:01:02.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=1 2024-09-15T06:01:02.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 1-98784247811 2024-09-15T06:01:02.073 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T06:01:02.074 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=98784247811 2024-09-15T06:01:02.074 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.1 seq 98784247811 2024-09-15T06:01:02.075 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-09-15T06:01:02.075 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-15T06:01:02.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 98784247811 -lt 98784247811 2024-09-15T06:01:02.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2181: flush_pg_stats: for s in $seqs 2024-09-15T06:01:02.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: echo 2-64424509453 2024-09-15T06:01:02.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: cut -d - -f 1 2024-09-15T06:01:02.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: flush_pg_stats: osd=2 2024-09-15T06:01:02.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: echo 2-64424509453 2024-09-15T06:01:02.430 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: cut -d - -f 2 2024-09-15T06:01:02.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: flush_pg_stats: seq=64424509453 2024-09-15T06:01:02.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-15T06:01:02.431 INFO:tasks.workunit.client.0.smithi138.stdout:waiting osd.2 seq 64424509453 2024-09-15T06:01:02.431 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-15T06:01:02.808 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-15T06:01:02.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-15T06:01:02.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T06:01:02.809 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T06:01:03.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-09-15T06:01:03.279 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T06:01:03.279 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T06:01:03.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T06:01:03.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T06:01:03.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T06:01:03.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T06:01:03.280 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T06:01:03.614 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T06:01:03.614 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T06:01:03.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T06:01:03.615 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T06:01:04.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T06:01:04.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-15T06:01:04.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-15T06:01:04.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-15T06:01:04.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T06:01:04.082 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T06:01:04.183 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T06:01:04.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-15T06:01:04.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T06:01:04.521 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T06:01:04.522 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T06:01:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-09-15T06:01:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-15T06:01:05.018 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-15T06:01:05.019 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-15T06:01:05.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-15T06:01:05.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-15T06:01:05.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-15T06:01:05.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-15T06:01:05.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-15T06:01:05.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-15T06:01:05.807 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-15T06:01:05.808 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-15T06:01:06.145 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-09-15T06:01:06.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-15T06:01:06.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-15T06:01:06.146 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-15T06:01:06.617 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:01:06.720 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836513 2024-09-15T06:01:06.742 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 98784247814 2024-09-15T06:01:06.762 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509456 2024-09-15T06:01:06.771 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:01:09.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:01:09.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:01:09.890 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:01:09.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:01:09.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:01:09.891 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-15T06:01:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:01:09.999 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:01:10.178 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:01:10.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:01:10.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:01:10.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-15T06:01:10.286 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:01:10.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:01:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:01:10.394 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:01:10.394 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:01:10.395 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:01:10.395 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:01:10.396 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:01:10.396 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:01:10.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:01:10.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:01:10.397 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:01:10.397 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:01:10.398 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:01:10.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T06:01:10.399 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:01:10.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:01:10.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:10.415 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:10.415 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:01:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:01:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:01:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-15T06:01:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-15T06:01:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:01:10.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:01:10.419 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:01:10.419 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:01:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:01:10.420 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:01:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:01:10.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:01:10.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:01:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:01:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:01:10.421 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:01:10.422 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:01:10.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:01:10.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T06:01:10.423 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:01:10.424 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:01:10.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:10.425 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:10.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:01:10.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:01:10.425 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:01:10.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-15T06:01:10.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T06:01:10.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:10.426 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:10.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.510179 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-15T06:01:10.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-15T06:01:10.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T06:01:10.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T06:01:10.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T06:01:10.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-15T06:01:10.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-15T06:01:10.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T06:01:10.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:01:10.789 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:01:10.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:01:10.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:10.790 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:10.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:10.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T06:01:11.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:11.040 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:11.041 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T06:01:11.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T06:01:11.041 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get fsid 2024-09-15T06:01:11.117 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T06:01:11.118 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T06:01:11.119 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get mon_host 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-15T06:01:11.192 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:12.025 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:12.026 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T06:01:12.026 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:01:12.050 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:12.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-15T06:01:12.052 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:01:12.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d245021b-5bfb-448f-ac63-57270ed0fa2b 2024-09-15T06:01:12.053 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d245021b-5bfb-448f-ac63-57270ed0fa2b' 2024-09-15T06:01:12.053 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 d245021b-5bfb-448f-ac63-57270ed0fa2b 2024-09-15T06:01:12.053 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:01:12.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAoeOZm/GTEAxAAulq6ioq+D380jwpz7gxbsw== 2024-09-15T06:01:12.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAoeOZm/GTEAxAAulq6ioq+D380jwpz7gxbsw=="}' 2024-09-15T06:01:12.064 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d245021b-5bfb-448f-ac63-57270ed0fa2b -i td/repro-long-log/0/new.json 2024-09-15T06:01:12.370 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:01:12.379 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-15T06:01:12.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoeOZm/GTEAxAAulq6ioq+D380jwpz7gxbsw== --osd-uuid d245021b-5bfb-448f-ac63-57270ed0fa2b 2024-09-15T06:01:12.402 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.401+0000 7f91a917d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:12.416 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.415+0000 7f91a917d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:12.417 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.416+0000 7f91a917d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:12.417 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.416+0000 7f91a917d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:01:12.417 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.416+0000 7f91a917d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:01:12.417 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.416+0000 7f91a917d440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:01:12.456 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:12.455+0000 7f91a917d440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-15T06:01:17.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-15T06:01:17.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:01:17.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T06:01:17.796 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T06:01:17.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:01:18.279 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T06:01:18.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T06:01:18.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:01:18.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:01:18.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:01:18.280 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:01:18.299 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:18.299+0000 7f27db616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:18.301 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:18.301+0000 7f27db616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:18.302 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:18.302+0000 7f27db616440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:18.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T06:01:18.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:01:18.625 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:18.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:18.820 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:18.820+0000 7f27db616440 -1 Falling back to public interface 2024-09-15T06:01:18.974 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:19.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:19.975 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:19.975 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:01:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:01:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:19.976 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:20.363 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:21.318 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:21.317+0000 7f27db616440 -1 osd.0 0 log_to_monitors true 2024-09-15T06:01:21.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:21.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:21.364 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:01:21.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:01:21.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:21.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:21.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:22.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:22.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:22.794 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:01:22.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:01:22.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:22.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:23.152 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:24.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:24.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:24.153 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:01:24.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:01:24.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:24.154 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:24.547 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:25.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:25.549 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:25.549 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:01:25.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:01:25.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:25.550 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:26.031 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:27.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:27.033 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:27.033 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:01:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:01:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:27.034 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:27.390 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:28.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:28.391 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:28.392 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:01:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:01:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:28.392 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:28.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:29.776 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:29.777 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:01:29.777 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:01:29.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:29.778 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:30.133 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:31.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:31.134 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:31.135 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:01:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:01:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:31.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:31.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:32.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:32.493 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:01:32.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:01:32.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:32.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:33.028 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:34.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:34.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:34.030 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:01:34.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:01:34.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:34.030 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:34.724 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:35.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:35.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:35.726 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:01:35.726 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:01:35.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:35.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:36.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:37.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:37.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:37.914 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:01:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:01:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:37.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:38.673 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:39.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:39.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:39.675 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:01:39.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:01:39.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:39.675 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:40.478 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:41.479 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:41.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:41.480 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:01:41.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:01:41.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:41.481 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:42.698 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:43.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:43.700 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:43.700 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:01:43.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:01:43.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:43.701 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:44.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:45.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:45.219 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:45.220 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:01:45.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:01:45.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:45.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:01:45.565 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/526628,v1:127.0.0.1:6803/526628] [v2:127.0.0.1:6804/526628,v1:127.0.0.1:6805/526628] exists,up d245021b-5bfb-448f-ac63-57270ed0fa2b 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:01:45.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:01:45.570 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:01:45.571 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02d1811b-271d-4668-afc4-24b36e35e156 2024-09-15T06:01:45.572 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 02d1811b-271d-4668-afc4-24b36e35e156' 2024-09-15T06:01:45.573 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 02d1811b-271d-4668-afc4-24b36e35e156 2024-09-15T06:01:45.573 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:01:45.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJeOZmWtyfIhAAjzhiarjcjNSsVRFklGntcg== 2024-09-15T06:01:45.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJeOZmWtyfIhAAjzhiarjcjNSsVRFklGntcg=="}' 2024-09-15T06:01:45.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02d1811b-271d-4668-afc4-24b36e35e156 -i td/repro-long-log/1/new.json 2024-09-15T06:01:46.112 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:01:46.123 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-15T06:01:46.124 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJeOZmWtyfIhAAjzhiarjcjNSsVRFklGntcg== --osd-uuid 02d1811b-271d-4668-afc4-24b36e35e156 2024-09-15T06:01:46.147 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.146+0000 7fc8aa94c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:46.148 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.148+0000 7fc8aa94c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:46.149 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.149+0000 7fc8aa94c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:46.150 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.149+0000 7fc8aa94c440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:01:46.150 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.149+0000 7fc8aa94c440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:01:46.150 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.149+0000 7fc8aa94c440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:01:46.213 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:46.212+0000 7fc8aa94c440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-15T06:01:51.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-15T06:01:51.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:01:51.796 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T06:01:51.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T06:01:51.797 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:01:52.307 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T06:01:52.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T06:01:52.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:01:52.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:01:52.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:01:52.308 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:01:52.328 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:52.327+0000 7f7038728440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:52.359 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:52.358+0000 7f7038728440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:52.361 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:52.359+0000 7f7038728440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:01:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T06:01:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:01:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T06:01:52.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:52.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:01:52.874 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:52.873+0000 7f7038728440 -1 Falling back to public interface 2024-09-15T06:01:53.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:54.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:54.037 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:54.037 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:01:54.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:01:54.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:54.038 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:01:54.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:55.330 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:01:55.329+0000 7f7038728440 -1 osd.1 0 log_to_monitors true 2024-09-15T06:01:55.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:55.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:55.416 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:01:55.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:01:55.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:55.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:01:56.049 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:57.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:57.051 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:57.051 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:01:57.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:01:57.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:57.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:01:57.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:58.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:58.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:58.438 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:01:58.438 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:01:58.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:58.439 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:01:58.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:01:59.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:01:59.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:01:59.821 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:01:59.821 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:01:59.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:01:59.822 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:00.228 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:01.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:01.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:01.230 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:02:01.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:02:01.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:01.230 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:01.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:02.634 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:02.634 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:02:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:02:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:02.635 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:03.090 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:04.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:04.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:04.092 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:02:04.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:02:04.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:04.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:04.606 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:05.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:05.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:05.608 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:02:05.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:02:05.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:05.609 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:06.114 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:07.115 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:07.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:07.116 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:02:07.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:02:07.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:07.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:07.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:08.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:08.473 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:08.473 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:02:08.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:02:08.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:08.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:08.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:09.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:09.865 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:09.865 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:02:09.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:02:09.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:09.866 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:10.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:11.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:11.905 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:11.906 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:02:11.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:02:11.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:11.906 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:12.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:13.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:13.660 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:13.660 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:02:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:02:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:13.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:14.484 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:15.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:15.486 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:15.486 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:02:15.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:02:15.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:15.487 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:16.368 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:17.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:17.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:17.369 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:02:17.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:02:17.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:17.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:18.157 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:19.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:19.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:19.159 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:02:19.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:02:19.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:19.159 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:02:19.550 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/528382,v1:127.0.0.1:6811/528382] [v2:127.0.0.1:6812/528382,v1:127.0.0.1:6813/528382] exists,up 02d1811b-271d-4668-afc4-24b36e35e156 2024-09-15T06:02:19.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:02:19.555 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:02:19.556 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:02:19.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86027e69-3858-4011-9465-73e95a2624fb 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 86027e69-3858-4011-9465-73e95a2624fb' 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 86027e69-3858-4011-9465-73e95a2624fb 2024-09-15T06:02:19.558 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:02:19.567 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBreOZmb63LIRAAhCczc0OD23PvL9qtr0NxUg== 2024-09-15T06:02:19.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBreOZmb63LIRAAhCczc0OD23PvL9qtr0NxUg=="}' 2024-09-15T06:02:19.568 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86027e69-3858-4011-9465-73e95a2624fb -i td/repro-long-log/2/new.json 2024-09-15T06:02:20.033 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:02:20.042 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-15T06:02:20.043 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBreOZmb63LIRAAhCczc0OD23PvL9qtr0NxUg== --osd-uuid 86027e69-3858-4011-9465-73e95a2624fb 2024-09-15T06:02:20.065 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.063+0000 7feb1de19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:20.067 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.065+0000 7feb1de19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:20.068 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.066+0000 7feb1de19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:20.069 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.067+0000 7feb1de19440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:02:20.069 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.067+0000 7feb1de19440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:02:20.069 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.067+0000 7feb1de19440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:02:20.114 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:20.113+0000 7feb1de19440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-15T06:02:25.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-15T06:02:25.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:02:25.925 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T06:02:25.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T06:02:25.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:02:26.403 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T06:02:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T06:02:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:02:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:02:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:02:26.404 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:02:26.424 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:26.423+0000 7f961272d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:26.434 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:26.433+0000 7f961272d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:26.435 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:26.434+0000 7f961272d440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:02:26.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:02:26.831 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:02:26.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:26.832 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:26.979 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:26.978+0000 7f961272d440 -1 Falling back to public interface 2024-09-15T06:02:27.178 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:28.179 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:28.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:28.180 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:02:28.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:02:28.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:28.180 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:28.533 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:29.445 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:02:29.445+0000 7f961272d440 -1 osd.2 0 log_to_monitors true 2024-09-15T06:02:29.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:29.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:29.534 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:02:29.535 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:02:29.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:29.535 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:29.977 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:30.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:30.978 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:30.978 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:02:30.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:02:30.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:30.979 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:31.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:32.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:02:32.494 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:02:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:32.495 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:32.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:33.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:33.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:33.858 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:02:33.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:02:33.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:33.859 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:34.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:35.332 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:35.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:35.333 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:02:35.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:02:35.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:35.333 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:35.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:36.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:36.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:36.769 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:02:36.769 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:02:36.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:36.770 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:37.251 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:38.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:38.252 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:38.252 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:02:38.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:02:38.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:38.253 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:38.762 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:39.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:39.764 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:39.764 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:02:39.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:02:39.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:39.765 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:40.195 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:41.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:41.197 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:02:41.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:02:41.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:41.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:41.782 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:42.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:42.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:42.784 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:02:42.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:02:42.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:42.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:43.370 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:44.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:44.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:44.372 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:02:44.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:02:44.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:44.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:44.810 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:45.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:45.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:45.812 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:02:45.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:02:45.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:45.813 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:46.522 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:47.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:47.524 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:47.524 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:02:47.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:02:47.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:47.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:48.095 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:49.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:49.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:49.096 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:02:49.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:02:49.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:49.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:49.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:50.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:50.794 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:50.794 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:02:50.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:02:50.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:50.795 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:51.647 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:02:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:02:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:02:52.649 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:02:52.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:02:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:02:52.650 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:02:53.009 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/530136,v1:127.0.0.1:6819/530136] [v2:127.0.0.1:6820/530136,v1:127.0.0.1:6821/530136] exists,up 86027e69-3858-4011-9465-73e95a2624fb 2024-09-15T06:02:53.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:02:53.009 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:02:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:02:53.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-15T06:02:53.626 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T06:02:53.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-15T06:02:53.637 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-15T06:02:54.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-15T06:02:54.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-15T06:02:54.223 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-15T06:02:54.367 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:02:54.392 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:02:54.417 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:02:54.426 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-15T06:02:54.541 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:02:54.566 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:02:54.591 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:02:54.600 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-15T06:02:54.696 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:02:54.721 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:02:54.746 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:02:54.755 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-15T06:02:54.852 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:02:54.877 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:02:54.901 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:02:54.910 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-15T06:02:55.018 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:02:55.043 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:02:55.068 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:02:55.076 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-15T06:02:55.077 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-15T06:02:55.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.078 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:55.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.302 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:55.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.411 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:55.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:55.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:55.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:55.879 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.021 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.122 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.255 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.256 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.356 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.457 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.674 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.775 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:56.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:56.917 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:57.025 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:57.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:57.227 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:57.327 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:02:57.427 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:02:57.569 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-15T06:02:57.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:02:57.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-15T06:02:57.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:02:57.570 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:02:57.664 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836502 2024-09-15T06:02:57.686 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672972 2024-09-15T06:02:57.707 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509446 2024-09-15T06:02:57.716 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:00.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:00.718 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:00.835 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:03:00.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:00.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:00.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-15T06:03:00.943 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:03:00.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:00.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:01.051 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:01.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:01.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:01.052 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:01.158 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:01.158 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-15T06:03:01.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.378 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.380 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.495 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.497 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.596 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.721 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.723 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.824 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.825 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:01.914 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:01.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:01.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.006 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.008 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.133 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.135 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.273 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.408 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.409 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.410 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.508 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.608 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.610 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.734 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.851 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:02.968 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:02.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:02.970 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:03.086 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:03.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:03.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:03.186 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:03.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:03.188 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:03.286 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:03.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:03.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:03.387 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:03.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:03:03.388 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:03:03.504 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:03.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-15T06:03:03.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:03.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-15T06:03:03.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:03.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:03.600 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836505 2024-09-15T06:03:03.622 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672975 2024-09-15T06:03:03.643 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509450 2024-09-15T06:03:03.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:06.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:06.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:06.773 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:03:06.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:06.773 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:06.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-15T06:03:06.881 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:03:06.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:06.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:06.988 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:06.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:07.095 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:07.096 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-15T06:03:07.192 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:03:07.217 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:03:07.241 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:03:07.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-15T06:03:07.347 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:03:07.372 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:03:07.396 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:03:07.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-15T06:03:07.502 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:03:07.528 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:03:07.552 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:03:07.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-15T06:03:07.677 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:03:07.702 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:03:07.727 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:03:07.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-09-15T06:03:07.834 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:03:07.859 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:03:07.885 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:03:07.894 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:08.059 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:08.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-09-15T06:03:08.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:08.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-15T06:03:08.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:08.061 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:08.167 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836508 2024-09-15T06:03:08.189 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672978 2024-09-15T06:03:08.211 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509453 2024-09-15T06:03:08.220 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:11.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:11.222 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:11.340 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:03:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:11.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-15T06:03:11.448 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:03:11.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:11.448 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:11.556 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:11.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:11.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:11.557 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:11.664 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:11.664 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:11.786 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:11.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-09-15T06:03:11.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:11.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-15T06:03:11.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:11.788 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:11.882 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836511 2024-09-15T06:03:11.904 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672981 2024-09-15T06:03:11.925 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509455 2024-09-15T06:03:11.934 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:14.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:14.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:15.054 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:03:15.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:15.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:15.055 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-15T06:03:15.162 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:03:15.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:15.162 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:15.269 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:15.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:15.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:15.270 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:15.378 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:15.378 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:15.504 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-09-15T06:03:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-15T06:03:15.506 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:15.507 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:15.601 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836514 2024-09-15T06:03:15.622 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672984 2024-09-15T06:03:15.643 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509458 2024-09-15T06:03:15.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:18.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:18.654 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:18.773 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:03:18.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:18.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:18.774 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-15T06:03:18.881 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:03:18.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:18.882 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:18.989 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:18.989 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:18.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:18.990 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:19.096 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:19.097 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:19.224 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-09-15T06:03:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-15T06:03:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:19.226 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:19.318 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836516 2024-09-15T06:03:19.340 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672986 2024-09-15T06:03:19.361 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509461 2024-09-15T06:03:19.369 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:22.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:22.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:22.489 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:03:22.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:22.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:22.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-15T06:03:22.598 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:03:22.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:22.599 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:22.706 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:22.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:22.706 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:22.707 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:22.812 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:22.812 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:22.933 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:22.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-09-15T06:03:22.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:22.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-15T06:03:22.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:22.936 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:23.029 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836519 2024-09-15T06:03:23.051 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672989 2024-09-15T06:03:23.072 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509464 2024-09-15T06:03:23.081 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:26.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:26.083 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:26.202 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:03:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:26.202 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-15T06:03:26.310 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:03:26.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:26.310 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:26.417 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:26.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:26.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:26.418 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:26.525 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:26.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:26.636 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:26.638 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-15T06:03:26.639 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:03:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:26.640 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:26.732 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836522 2024-09-15T06:03:26.753 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672992 2024-09-15T06:03:26.774 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509466 2024-09-15T06:03:26.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:29.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:29.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:29.903 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:29.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:29.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:29.904 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:03:30.011 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:30.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:30.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:30.120 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:30.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:30.260 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:30.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:30.362 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:30.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-15T06:03:30.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:30.364 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-15T06:03:30.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:30.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:30.459 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836525 2024-09-15T06:03:30.480 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672995 2024-09-15T06:03:30.501 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509469 2024-09-15T06:03:30.510 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:33.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:33.512 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:33.631 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:03:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:33.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-15T06:03:33.740 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:03:33.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:33.741 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:33.847 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:33.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:33.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:33.848 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:33.955 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:33.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:34.114 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:34.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-15T06:03:34.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:34.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:03:34.116 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:34.117 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:34.232 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836527 2024-09-15T06:03:34.254 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672997 2024-09-15T06:03:34.275 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509472 2024-09-15T06:03:34.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:37.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:37.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:37.404 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:37.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:37.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:37.405 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:03:37.512 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:37.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:37.513 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:37.619 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:37.620 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:37.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:37.621 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:37.726 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:37.727 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:37.858 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:37.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-09-15T06:03:37.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:37.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-15T06:03:37.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:37.860 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:37.954 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836530 2024-09-15T06:03:37.976 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949673000 2024-09-15T06:03:37.997 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509475 2024-09-15T06:03:38.005 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:41.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:41.007 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:41.125 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:03:41.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:41.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:41.126 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-15T06:03:41.233 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:03:41.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:41.234 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:41.340 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:41.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:41.340 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:41.341 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:41.446 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:41.447 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-09-15T06:03:41.560 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:03:41.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-09-15T06:03:41.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:03:41.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:03:41.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:03:41.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:03:41.656 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836533 2024-09-15T06:03:41.678 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949673003 2024-09-15T06:03:41.700 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509477 2024-09-15T06:03:41.709 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:03:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:44.710 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:44.829 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:44.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:44.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:03:44.830 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:03:44.937 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:44.937 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:44.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:45.044 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:45.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:03:45.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:03:45.045 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:03:45.151 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:03:45.153 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:03:45.259 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:03:45.260 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:03:45.260 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:03:45.261 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:03:45.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:03:45.262 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:03:45.262 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:03:45.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:03:45.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:03:45.263 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:03:45.263 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:03:45.264 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:03:45.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T06:03:45.265 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:03:45.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:03:45.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.282 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.282 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:03:45.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:03:45.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:03:45.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:03:45.285 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:03:45.285 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:03:45.286 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:03:45.286 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:03:45.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:03:45.287 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:03:45.287 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:03:45.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:03:45.288 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:03:45.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:03:45.289 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:03:45.289 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:03:45.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T06:03:45.290 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:03:45.291 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:03:45.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.292 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:03:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:03:45.292 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:03:45.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-09-15T06:03:45.293 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-15T06:03:45.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.294 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.294 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.510179 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-09-15T06:03:45.296 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-09-15T06:03:45.297 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:03:45.652 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-15T06:03:45.920 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-15T06:03:45.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-15T06:03:45.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T06:03:45.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T06:03:45.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-15T06:03:45.921 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T06:03:45.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T06:03:45.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get fsid 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-15T06:03:45.998 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-15T06:03:45.999 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:45.999 INFO:tasks.workunit.client.0.smithi138.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:45.999 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.510179/ceph-mon.a.asok 2024-09-15T06:03:45.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-15T06:03:45.999 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.510179/ceph-mon.a.asok config get mon_host 2024-09-15T06:03:46.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-09-15T06:03:46.073 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:46.915 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:03:46.916 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T06:03:46.916 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-15T06:03:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-09-15T06:03:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:03:46.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:03:46.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:03:46.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:03:46.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ab7dd35-69d1-4190-b7a0-d094a5ea5be6 2024-09-15T06:03:46.944 INFO:tasks.workunit.client.0.smithi138.stdout:add osd0 4ab7dd35-69d1-4190-b7a0-d094a5ea5be6 2024-09-15T06:03:46.944 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4ab7dd35-69d1-4190-b7a0-d094a5ea5be6' 2024-09-15T06:03:46.944 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:03:46.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCeOZmgWjPOBAAMRsANBg4GDWWauHte8C8Rg== 2024-09-15T06:03:46.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCeOZmgWjPOBAAMRsANBg4GDWWauHte8C8Rg=="}' 2024-09-15T06:03:46.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ab7dd35-69d1-4190-b7a0-d094a5ea5be6 -i td/repro-long-log/0/new.json 2024-09-15T06:03:47.259 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:47.268 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-09-15T06:03:47.269 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCeOZmgWjPOBAAMRsANBg4GDWWauHte8C8Rg== --osd-uuid 4ab7dd35-69d1-4190-b7a0-d094a5ea5be6 2024-09-15T06:03:47.292 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.291+0000 7f94b7c19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:47.304 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.303+0000 7f94b7c19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:47.305 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.305+0000 7f94b7c19440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:47.306 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.305+0000 7f94b7c19440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:03:47.306 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.305+0000 7f94b7c19440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:03:47.306 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.305+0000 7f94b7c19440 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-09-15T06:03:47.353 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:47.352+0000 7f94b7c19440 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-09-15T06:03:52.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-09-15T06:03:52.412 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:03:52.412 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd0 key to auth repository 2024-09-15T06:03:52.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-15T06:03:52.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:03:52.910 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.0 2024-09-15T06:03:52.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-15T06:03:52.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:03:52.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:03:52.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:03:52.911 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:03:52.931 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:52.929+0000 7f39ea588440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:52.958 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:52.956+0000 7f39ea588440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:52.959 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:52.957+0000 7f39ea588440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:03:53.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-15T06:03:53.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:03:53.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-15T06:03:53.305 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:03:53.305 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:03:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:03:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:03:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:03:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:03:53.306 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:03:53.489 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:53.488+0000 7f39ea588440 -1 Falling back to public interface 2024-09-15T06:03:53.713 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:03:54.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:03:54.714 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:03:54.714 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:03:54.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:03:54.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:03:54.715 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:03:55.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:03:55.902 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:03:55.902+0000 7f39ea588440 -1 osd.0 0 log_to_monitors true 2024-09-15T06:03:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:03:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:03:56.062 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:03:56.062 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:03:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:03:56.063 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:03:56.463 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:03:57.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:03:57.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:03:57.465 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:03:57.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:03:57.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:03:57.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:03:57.994 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:03:58.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:03:58.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:03:58.996 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:03:58.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:03:58.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:03:58.997 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:03:59.371 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:00.372 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:00.373 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:04:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:04:00.373 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:00.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:00.743 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:01.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:01.744 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:01.744 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:04:01.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:04:01.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:01.745 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:02.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:03.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:03.130 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:04:03.130 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:04:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:03.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:03.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:04.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:04.607 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:04.607 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:04:04.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:04:04.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:04.608 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:04.969 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:05.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:05.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:05.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:04:05.971 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:04:05.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:05.972 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:06.374 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:07.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:07.375 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:07.375 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:04:07.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:04:07.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:07.376 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:08.212 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:09.213 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:09.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:09.214 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:04:09.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:04:09.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:09.214 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:09.861 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:10.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:10.862 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:10.862 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:04:10.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:04:10.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:10.863 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:12.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:13.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:13.058 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:13.059 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:04:13.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:04:13.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:13.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:13.559 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:14.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:14.561 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:14.561 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:04:14.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:04:14.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:14.562 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:15.856 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:16.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:16.857 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:16.857 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:04:16.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:04:16.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:16.858 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:17.783 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:18.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:18.784 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:18.784 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:04:18.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:04:18.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:18.785 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:19.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:20.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:20.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:20.246 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:04:20.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:04:20.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:20.247 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-15T06:04:20.625 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/535088,v1:127.0.0.1:6803/535088] [v2:127.0.0.1:6804/535088,v1:127.0.0.1:6805/535088] exists,up 4ab7dd35-69d1-4190-b7a0-d094a5ea5be6 2024-09-15T06:04:20.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:04:20.630 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:04:20.631 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-09-15T06:04:20.632 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:04:20.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=257e5559-70ea-4394-9b07-339b0cefc28f 2024-09-15T06:04:20.633 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 257e5559-70ea-4394-9b07-339b0cefc28f' 2024-09-15T06:04:20.634 INFO:tasks.workunit.client.0.smithi138.stdout:add osd1 257e5559-70ea-4394-9b07-339b0cefc28f 2024-09-15T06:04:20.634 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:04:20.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkeOZmlgQ8JhAAwZm1xWnLSw93CocgQp7WNg== 2024-09-15T06:04:20.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkeOZmlgQ8JhAAwZm1xWnLSw93CocgQp7WNg=="}' 2024-09-15T06:04:20.642 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 257e5559-70ea-4394-9b07-339b0cefc28f -i td/repro-long-log/1/new.json 2024-09-15T06:04:21.068 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:04:21.079 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-09-15T06:04:21.080 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkeOZmlgQ8JhAAwZm1xWnLSw93CocgQp7WNg== --osd-uuid 257e5559-70ea-4394-9b07-339b0cefc28f 2024-09-15T06:04:21.102 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.101+0000 7f8b98a8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:21.148 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.147+0000 7f8b98a8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:21.152 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.148+0000 7f8b98a8b440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:21.152 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.149+0000 7f8b98a8b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:04:21.152 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.149+0000 7f8b98a8b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:04:21.152 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.149+0000 7f8b98a8b440 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-09-15T06:04:21.188 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:21.187+0000 7f8b98a8b440 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-09-15T06:04:26.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-09-15T06:04:26.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:04:26.991 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-15T06:04:26.992 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd1 key to auth repository 2024-09-15T06:04:26.992 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:04:27.493 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.1 2024-09-15T06:04:27.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-15T06:04:27.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:04:27.493 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:04:27.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:04:27.494 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:04:27.513 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:27.512+0000 7fc8a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:27.515 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:27.514+0000 7fc8a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:27.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:27.515+0000 7fc8a260e440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:27.836 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:27.837 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:04:27.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:04:27.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:27.838 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:28.032 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:28.031+0000 7fc8a260e440 -1 Falling back to public interface 2024-09-15T06:04:28.184 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:29.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:29.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:29.186 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:04:29.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:04:29.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:29.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:29.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:30.516 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:30.515+0000 7fc8a260e440 -1 osd.1 0 log_to_monitors true 2024-09-15T06:04:30.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:30.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:30.540 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:04:30.540 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:04:30.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:30.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:30.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:31.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:31.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:31.925 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:04:31.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:04:31.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:31.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:32.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:33.322 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:33.322 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:04:33.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:04:33.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:33.323 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:33.686 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:34.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:34.687 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:34.687 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:04:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:04:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:34.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:35.127 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:36.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:36.128 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:36.128 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:04:36.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:04:36.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:36.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:36.552 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:37.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:37.553 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:37.554 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:04:37.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:04:37.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:37.554 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:37.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:38.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:38.963 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:38.963 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:04:38.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:04:38.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:38.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:39.489 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:40.490 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:40.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:40.491 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:04:40.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:04:40.491 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:40.492 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:40.851 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:41.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:41.852 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:41.852 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:04:41.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:04:41.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:41.853 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:42.472 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:43.474 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:04:43.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:04:43.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:43.475 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:44.139 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:45.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:45.141 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:04:45.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:04:45.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:45.142 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:45.802 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:46.804 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:46.804 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:04:46.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:04:46.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:46.805 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:47.578 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:48.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:48.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:48.580 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:04:48.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:04:48.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:48.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:49.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:50.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:50.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:50.318 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:04:50.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:04:50.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:50.319 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:50.888 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:51.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:51.889 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:51.889 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:04:51.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:04:51.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:51.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:52.580 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:04:53.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:04:53.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:04:53.581 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:04:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:04:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:04:53.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-15T06:04:53.943 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/536842,v1:127.0.0.1:6811/536842] [v2:127.0.0.1:6812/536842,v1:127.0.0.1:6813/536842] exists,up 257e5559-70ea-4394-9b07-339b0cefc28f 2024-09-15T06:04:53.948 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-15T06:04:53.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-15T06:04:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-15T06:04:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-09-15T06:04:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-09-15T06:04:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-15T06:04:53.950 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-09-15T06:04:53.951 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' 2024-09-15T06:04:53.952 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-15T06:04:53.953 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ed01623-db63-4af1-8362-afaabd0dd8e5 2024-09-15T06:04:53.954 INFO:tasks.workunit.client.0.smithi138.stdout:add osd2 6ed01623-db63-4af1-8362-afaabd0dd8e5 2024-09-15T06:04:53.955 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6ed01623-db63-4af1-8362-afaabd0dd8e5' 2024-09-15T06:04:53.955 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-15T06:04:53.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFeeZmQRQjORAA/YBcUhtF1WwmAl4mzzfJog== 2024-09-15T06:04:53.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFeeZmQRQjORAA/YBcUhtF1WwmAl4mzzfJog=="}' 2024-09-15T06:04:53.961 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ed01623-db63-4af1-8362-afaabd0dd8e5 -i td/repro-long-log/2/new.json 2024-09-15T06:04:54.402 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:04:54.413 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-09-15T06:04:54.414 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFeeZmQRQjORAA/YBcUhtF1WwmAl4mzzfJog== --osd-uuid 6ed01623-db63-4af1-8362-afaabd0dd8e5 2024-09-15T06:04:54.437 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.436+0000 7fdf50b86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:54.438 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.437+0000 7fdf50b86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:54.439 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.438+0000 7fdf50b86440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:04:54.439 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.439+0000 7fdf50b86440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:04:54.440 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.439+0000 7fdf50b86440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:04:54.440 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.439+0000 7fdf50b86440 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-09-15T06:04:54.501 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:04:54.500+0000 7fdf50b86440 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-09-15T06:05:00.590 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-09-15T06:05:00.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-15T06:05:00.591 INFO:tasks.workunit.client.0.smithi138.stdout:adding osd2 key to auth repository 2024-09-15T06:05:00.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-15T06:05:00.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-15T06:05:01.070 INFO:tasks.workunit.client.0.smithi138.stdout:start osd.2 2024-09-15T06:05:01.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-15T06:05:01.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=f4811be9-d729-4381-9edd-931123c1a1e6 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.510179/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-15T06:05:01.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-15T06:05:01.071 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-15T06:05:01.072 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-15T06:05:01.091 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:05:01.090+0000 7f1acf20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:05:01.093 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:05:01.092+0000 7f1acf20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:05:01.094 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:05:01.093+0000 7f1acf20c440 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-15T06:05:01.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-15T06:05:01.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-15T06:05:01.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-15T06:05:01.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:01.469 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:01.608 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:05:01.607+0000 7f1acf20c440 -1 Falling back to public interface 2024-09-15T06:05:01.816 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:02.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:02.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:02.818 INFO:tasks.workunit.client.0.smithi138.stdout:1 2024-09-15T06:05:02.818 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-15T06:05:02.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:02.819 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:03.216 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:04.122 INFO:tasks.workunit.client.0.smithi138.stderr:2024-09-15T06:05:04.121+0000 7f1acf20c440 -1 osd.2 0 log_to_monitors true 2024-09-15T06:05:04.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:04.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:04.217 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-15T06:05:04.217 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:05:04.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:04.218 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:04.678 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:05.679 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:05.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:05.680 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:05:05.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-15T06:05:05.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:05.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:06.316 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:07.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:07.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:07.317 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-15T06:05:07.318 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:05:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:07.318 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:07.734 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:08.735 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:08.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:08.736 INFO:tasks.workunit.client.0.smithi138.stdout:5 2024-09-15T06:05:08.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-15T06:05:08.736 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:08.737 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:09.092 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:10.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:10.093 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:10.094 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:05:10.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-15T06:05:10.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:10.094 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:10.480 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-15T06:05:11.482 INFO:tasks.workunit.client.0.smithi138.stdout:7 2024-09-15T06:05:11.482 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:11.483 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:11.938 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:12.939 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:12.940 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:12.940 INFO:tasks.workunit.client.0.smithi138.stdout:8 2024-09-15T06:05:12.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-15T06:05:12.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:12.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:13.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:14.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:14.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:14.436 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-15T06:05:14.436 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:05:14.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:14.437 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:14.913 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:15.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:15.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:15.915 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:05:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-15T06:05:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:15.915 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:16.539 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:17.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:17.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:17.541 INFO:tasks.workunit.client.0.smithi138.stdout:11 2024-09-15T06:05:17.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-15T06:05:17.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:17.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:18.594 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:19.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:19.596 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:19.597 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:05:19.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-15T06:05:19.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:19.597 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:20.086 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:21.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:21.087 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:21.088 INFO:tasks.workunit.client.0.smithi138.stdout:13 2024-09-15T06:05:21.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-15T06:05:21.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:21.088 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:22.197 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:23.198 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:23.199 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:23.199 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:05:23.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-15T06:05:23.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:23.200 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:23.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:24.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:24.928 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:24.929 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:05:24.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-15T06:05:24.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:24.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:25.747 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:26.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:26.748 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:26.749 INFO:tasks.workunit.client.0.smithi138.stdout:16 2024-09-15T06:05:26.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-15T06:05:26.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:26.749 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:27.274 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-15T06:05:28.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-15T06:05:28.275 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-15T06:05:28.275 INFO:tasks.workunit.client.0.smithi138.stdout:17 2024-09-15T06:05:28.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-15T06:05:28.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-15T06:05:28.276 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-15T06:05:28.627 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/538597,v1:127.0.0.1:6819/538597] [v2:127.0.0.1:6820/538597,v1:127.0.0.1:6821/538597] exists,up 6ed01623-db63-4af1-8362-afaabd0dd8e5 2024-09-15T06:05:28.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-15T06:05:28.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-15T06:05:28.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-15T06:05:28.628 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-09-15T06:05:29.149 INFO:tasks.workunit.client.0.smithi138.stderr:pool 'test' already exists 2024-09-15T06:05:29.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-09-15T06:05:29.160 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-09-15T06:05:29.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-09-15T06:05:29.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-09-15T06:05:29.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-09-15T06:05:29.617 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:29.642 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:29.667 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:29.676 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-09-15T06:05:29.783 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:29.809 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:29.834 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:29.843 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-09-15T06:05:29.969 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:29.995 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:30.020 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:30.029 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-09-15T06:05:30.185 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:30.210 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:30.235 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:30.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-15T06:05:30.362 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:30.388 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:30.412 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:30.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-09-15T06:05:30.421 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-09-15T06:05:30.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:30.422 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:31.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:31.458 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:31.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:31.667 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:31.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:31.885 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.035 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.136 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.245 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.337 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.429 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.529 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.680 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:32.880 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:32.881 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.023 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.140 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.141 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.266 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.383 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.509 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.626 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:33.768 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-09-15T06:05:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-09-15T06:05:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-09-15T06:05:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:05:33.877 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:33.971 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836501 2024-09-15T06:05:33.992 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672975 2024-09-15T06:05:34.014 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509447 2024-09-15T06:05:34.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:37.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:37.024 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:37.143 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:05:37.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:37.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:37.144 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-09-15T06:05:37.250 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:05:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:37.250 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:37.356 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:05:37.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:37.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:37.357 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:05:37.464 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:05:37.464 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-09-15T06:05:37.604 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-09-15T06:05:37.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:37.605 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:37.703 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:37.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:37.705 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:37.812 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:37.814 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:37.912 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:37.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:37.914 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.012 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.014 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.130 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.131 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.247 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.249 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.414 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.416 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.590 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.591 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.592 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.740 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.742 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:38.882 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:38.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:38.884 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.008 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.010 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.150 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.159 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.160 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.161 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.318 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.320 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.418 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.420 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.519 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.520 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.521 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.628 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.629 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.736 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.738 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.845 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.846 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.847 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:39.962 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:39.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-09-15T06:05:39.964 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-09-15T06:05:40.096 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:40.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-09-15T06:05:40.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:40.098 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-09-15T06:05:40.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-09-15T06:05:40.099 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:40.192 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836504 2024-09-15T06:05:40.214 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672978 2024-09-15T06:05:40.235 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509450 2024-09-15T06:05:40.244 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:43.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:43.246 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:43.364 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:05:43.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:43.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:43.365 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-09-15T06:05:43.473 INFO:tasks.workunit.client.0.smithi138.stdout:21 2024-09-15T06:05:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:43.474 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:43.581 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:05:43.581 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:43.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:43.582 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-09-15T06:05:43.688 INFO:tasks.workunit.client.0.smithi138.stdout:0 2024-09-15T06:05:43.688 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-09-15T06:05:43.786 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:43.811 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:43.836 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:43.844 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-09-15T06:05:43.940 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:43.965 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:43.989 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:43.998 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-09-15T06:05:44.096 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:44.122 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:44.147 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:44.156 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-09-15T06:05:44.254 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:44.279 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:44.303 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:44.312 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-09-15T06:05:44.410 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: {} 2024-09-15T06:05:44.435 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: {} 2024-09-15T06:05:44.460 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: {} 2024-09-15T06:05:44.468 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:05:44.644 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:44.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-09-15T06:05:44.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:44.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-09-15T06:05:44.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-09-15T06:05:44.646 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:44.739 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836507 2024-09-15T06:05:44.761 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672981 2024-09-15T06:05:44.782 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509453 2024-09-15T06:05:44.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:47.792 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:47.793 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:47.911 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:05:47.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:47.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:47.912 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-09-15T06:05:48.021 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:05:48.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:48.022 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:48.128 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:05:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:48.129 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-09-15T06:05:48.236 INFO:tasks.workunit.client.0.smithi138.stdout:2 2024-09-15T06:05:48.236 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:05:48.379 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:48.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-09-15T06:05:48.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:48.381 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-09-15T06:05:48.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-09-15T06:05:48.382 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:48.477 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836510 2024-09-15T06:05:48.498 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672983 2024-09-15T06:05:48.520 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509455 2024-09-15T06:05:48.528 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:51.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:51.530 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:51.648 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:05:51.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:51.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:51.649 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-09-15T06:05:51.756 INFO:tasks.workunit.client.0.smithi138.stdout:15 2024-09-15T06:05:51.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:51.757 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:51.863 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:05:51.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:51.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:51.864 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-09-15T06:05:51.970 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:05:51.971 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:05:52.057 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-09-15T06:05:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-09-15T06:05:52.059 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-09-15T06:05:52.060 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:52.153 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836513 2024-09-15T06:05:52.175 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672986 2024-09-15T06:05:52.196 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509458 2024-09-15T06:05:52.204 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:55.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:55.206 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:55.325 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:05:55.325 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:55.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:55.326 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-09-15T06:05:55.433 INFO:tasks.workunit.client.0.smithi138.stdout:12 2024-09-15T06:05:55.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:55.434 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:55.541 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:05:55.541 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:55.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:55.542 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-09-15T06:05:55.648 INFO:tasks.workunit.client.0.smithi138.stdout:10 2024-09-15T06:05:55.648 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:05:55.794 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:55.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-09-15T06:05:55.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:55.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-09-15T06:05:55.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-09-15T06:05:55.796 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:55.890 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836516 2024-09-15T06:05:55.911 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672989 2024-09-15T06:05:55.932 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509461 2024-09-15T06:05:55.941 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:05:58.942 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:58.943 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:59.076 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:05:59.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:59.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:05:59.077 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-09-15T06:05:59.184 INFO:tasks.workunit.client.0.smithi138.stdout:9 2024-09-15T06:05:59.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:59.185 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:59.292 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:05:59.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:05:59.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:05:59.293 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-09-15T06:05:59.399 INFO:tasks.workunit.client.0.smithi138.stdout:14 2024-09-15T06:05:59.400 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:05:59.512 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:05:59.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-09-15T06:05:59.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:05:59.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-09-15T06:05:59.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-09-15T06:05:59.514 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:05:59.607 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836518 2024-09-15T06:05:59.629 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672992 2024-09-15T06:05:59.650 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509464 2024-09-15T06:05:59.659 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:02.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:02.661 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:02.778 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:06:02.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:02.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:02.779 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-09-15T06:06:02.887 INFO:tasks.workunit.client.0.smithi138.stdout:6 2024-09-15T06:06:02.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:02.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:02.995 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:06:02.995 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:02.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:02.996 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-09-15T06:06:03.146 INFO:tasks.workunit.client.0.smithi138.stdout:18 2024-09-15T06:06:03.146 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:06:03.281 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:06:03.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-15T06:06:03.283 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:06:03.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:06:03.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-15T06:06:03.284 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:06:03.377 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836521 2024-09-15T06:06:03.399 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672994 2024-09-15T06:06:03.420 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509466 2024-09-15T06:06:03.428 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:06.430 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:06.431 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:06.564 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:06.564 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:06.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:06.565 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:06:06.672 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:06.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:06.672 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:06.779 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:06.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:06.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:06.780 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-15T06:06:06.887 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:06.887 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:06:07.033 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:06:07.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-15T06:06:07.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:06:07.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-15T06:06:07.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-15T06:06:07.036 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:06:07.136 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836524 2024-09-15T06:06:07.157 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949672997 2024-09-15T06:06:07.178 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509469 2024-09-15T06:06:07.186 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:10.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:10.189 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:10.308 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:06:10.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:10.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:10.309 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-15T06:06:10.416 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:06:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:10.417 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:10.524 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:10.525 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-15T06:06:10.631 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:10.632 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:06:10.744 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:06:10.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-15T06:06:10.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:06:10.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:06:10.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-15T06:06:10.746 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:06:10.839 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836527 2024-09-15T06:06:10.860 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949673000 2024-09-15T06:06:10.881 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509472 2024-09-15T06:06:10.890 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:13.892 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:13.893 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:14.012 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:14.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:14.012 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:14.013 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:06:14.120 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:14.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:14.121 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:14.229 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:14.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:14.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:14.229 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-15T06:06:14.337 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:14.338 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:06:14.463 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:06:14.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-09-15T06:06:14.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:06:14.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-09-15T06:06:14.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-15T06:06:14.465 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:06:14.560 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836529 2024-09-15T06:06:14.581 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949673003 2024-09-15T06:06:14.603 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509475 2024-09-15T06:06:14.611 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:17.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:17.613 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:17.732 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:06:17.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:17.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:17.733 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-09-15T06:06:17.840 INFO:tasks.workunit.client.0.smithi138.stdout:4 2024-09-15T06:06:17.840 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:17.841 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:17.948 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:17.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:17.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:17.949 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-15T06:06:18.057 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:18.057 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-09-15T06:06:18.199 INFO:tasks.workunit.client.0.smithi138.stderr:error removing test>foo: (2) No such file or directory 2024-09-15T06:06:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-09-15T06:06:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-09-15T06:06:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-09-15T06:06:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-09-15T06:06:18.201 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-09-15T06:06:18.294 INFO:tasks.workunit.client.0.smithi138.stdout:osd.0: 21474836532 2024-09-15T06:06:18.316 INFO:tasks.workunit.client.0.smithi138.stdout:osd.1: 42949673005 2024-09-15T06:06:18.337 INFO:tasks.workunit.client.0.smithi138.stdout:osd.2: 64424509477 2024-09-15T06:06:18.345 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-09-15T06:06:21.347 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:21.348 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:21.466 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:21.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:21.466 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-09-15T06:06:21.467 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-09-15T06:06:21.573 INFO:tasks.workunit.client.0.smithi138.stdout:3 2024-09-15T06:06:21.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:21.574 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:21.680 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:21.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-09-15T06:06:21.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-09-15T06:06:21.681 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-09-15T06:06:21.788 INFO:tasks.workunit.client.0.smithi138.stdout:20 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:06:21.790 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:06:21.897 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:06:21.897 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:06:21.898 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:06:21.898 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:06:21.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:06:21.899 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:06:21.899 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:06:21.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:06:21.900 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:06:21.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:06:21.901 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:06:21.901 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:06:21.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-15T06:06:21.902 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:06:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:06:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:06:21.919 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:06:21.919 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: main: code=0 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: main: teardown td/repro-long-log 0 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-09-15T06:06:21.921 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-15T06:06:21.922 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-15T06:06:21.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-15T06:06:21.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-15T06:06:21.923 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-15T06:06:21.923 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-15T06:06:21.924 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-15T06:06:21.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-15T06:06:21.925 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-15T06:06:21.925 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:06:21.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-15T06:06:21.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-15T06:06:21.926 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T06:06:21.926 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:06:21.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-15T06:06:21.927 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-09-15T06:06:21.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-15T06:06:21.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-15T06:06:21.929 INFO:tasks.workunit.client.0.smithi138.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.510179 2024-09-15T06:06:21.929 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.510179 2024-09-15T06:06:21.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-15T06:06:21.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-15T06:06:21.930 INFO:tasks.workunit.client.0.smithi138.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: main: return 0 2024-09-15T06:06:21.931 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T06:06:21.931 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T06:06:21.955 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-09-15T06:06:21.955 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T06:06:22.265 DEBUG:teuthology.parallel:result is None 2024-09-15T06:06:22.265 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T06:06:22.289 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T06:06:22.289 DEBUG:teuthology.orchestra.run.smithi138:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T06:06:22.353 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T06:06:22.353 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T06:06:22.367 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T06:06:22.367 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T06:06:22.442 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T06:06:22.457 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T06:06:22.457 DEBUG:teuthology.orchestra.run.smithi138:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T06:06:22.471 INFO:teuthology.orchestra.run.smithi138.stderr:bash: line 1: ntpq: command not found 2024-09-15T06:06:22.476 INFO:teuthology.orchestra.run.smithi138.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-15T06:06:22.476 INFO:teuthology.orchestra.run.smithi138.stdout:=============================================================================== 2024-09-15T06:06:22.476 INFO:teuthology.orchestra.run.smithi138.stdout:^* hv01.front.sepia.ceph.com 3 7 377 53 -104us[ -43us] +/- 39ms 2024-09-15T06:06:22.476 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 126 -251us[ -191us] +/- 67ms 2024-09-15T06:06:22.477 INFO:teuthology.orchestra.run.smithi138.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 +498us[ +498us] +/- 60ms 2024-09-15T06:06:22.477 INFO:teuthology.orchestra.run.smithi138.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-09-15T06:06:22.477 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T06:06:22.501 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T06:06:22.501 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T06:06:22.509 DEBUG:teuthology.orchestra.run.smithi138:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-15T06:06:22.575 DEBUG:teuthology.orchestra.run.smithi138:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-15T06:06:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T06:06:22.602 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T06:06:22.614 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T06:06:22.622 INFO:teuthology.task.internal:Duration was 9209.123636 seconds 2024-09-15T06:06:22.623 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T06:06:22.630 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T06:06:22.631 DEBUG:teuthology.orchestra.run.smithi138:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T06:06:22.680 INFO:teuthology.orchestra.run.smithi138.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-15T06:06:22.890 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T06:06:22.890 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi138.front.sepia.ceph.com 2024-09-15T06:06:22.891 DEBUG:teuthology.orchestra.run.smithi138:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T06:06:22.955 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T06:06:22.955 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T06:06:23.022 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T06:06:23.022 DEBUG:teuthology.orchestra.run.smithi138:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T06:06:23.168 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T06:06:23.177 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T06:06:23.177 DEBUG:teuthology.orchestra.run.smithi138:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T06:06:23.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T06:06:23.314 DEBUG:teuthology.orchestra.run.smithi138:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:06:23.337 INFO:teuthology.orchestra.run.smithi138.stdout:kernel.core_pattern = core 2024-09-15T06:06:23.350 DEBUG:teuthology.orchestra.run.smithi138:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T06:06:23.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T06:06:23.405 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T06:06:23.415 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T06:06:23.417 DEBUG:teuthology.misc:Transferring archived files from smithi138:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-15_00:54:39-rados-wip-bharath9-testing-2024-09-13-1444-quincy-distro-default-smithi/7905297/remote/smithi138 2024-09-15T06:06:23.418 DEBUG:teuthology.orchestra.run.smithi138:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T06:06:23.505 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T06:06:23.505 DEBUG:teuthology.orchestra.run.smithi138:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T06:06:23.520 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T06:06:23.530 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T06:06:23.530 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T06:06:23.539 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T06:06:23.539 DEBUG:teuthology.orchestra.run.smithi138:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T06:06:23.574 INFO:teuthology.orchestra.run.smithi138.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 06:06 /home/ubuntu/cephtest 2024-09-15T06:06:23.575 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T06:06:23.585 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T06:06:23.608 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{centos_latest} workloads/osd} duration: 9209.12363576889 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-15T06:06:23.608 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T06:06:23.696 INFO:teuthology.run:pass